summary refs log tree commit diff
path: root/tests/replication (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1610-89/+83
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-16/+25
|
* lintRichard van der Hoff2020-12-152-4/+1
|
* Remove spurious "SynapseRequest" result from `make_request"Richard van der Hoff2020-12-154-24/+24
| | | | This was never used, so let's get rid of it.
* Replace `request.code` with `channel.code`Richard van der Hoff2020-12-152-9/+9
| | | | | | The two are equivalent, but really we want to check the HTTP result that got returned to the channel, not the code that the Request object *intended* to return to the channel.
* Default to blacklisting reserved IP ranges and add a whitelist. (#8870)Patrick Cloke2020-12-091-1/+1
| | | | This defaults `ip_range_blacklist` to reserved IP ranges and also adds an `ip_range_whitelist` setting to override it.
* Add authentication to replication endpoints. (#8853)Patrick Cloke2020-12-042-8/+120
| | | | Authentication is done by checking a shared secret provided in the Synapse configuration file.
* Add additional validation to pusher URLs. (#8865)Patrick Cloke2020-12-041-4/+4
| | | | Pusher URLs now must end in `/_matrix/push/v1/notify` per the specification.
* Merge pull request #8858 from matrix-org/rav/sso_uiaRichard van der Hoff2020-12-021-6/+8
|\ | | | | UIA: offer only available auth flows
| * fix up various test casesRichard van der Hoff2020-12-021-6/+8
| | | | | | | | | | A few test cases were relying on being able to mount non-client servlets on the test resource. it's better to give them their own Resources.
* | Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-023-10/+10
|/ | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Simplify the way the `HomeServer` object caches its internal attributes. ↵Jonathan de Jong2020-11-301-1/+1
| | | | | (#8565) Changes `@cache_in_self` to use underscore-prefixed attributes.
* Remove redundant calls to `render()`Richard van der Hoff2020-11-163-16/+1
|
* Make `make_request` actually render the requestRichard van der Hoff2020-11-161-2/+2
| | | | | | remove the stubbing out of `request.process`, so that `requestReceived` also renders the request via the appropriate resource. Replace render() with a stub for now.
* Merge branch 'develop' into rav/pass_site_to_make_requestRichard van der Hoff2020-11-161-2/+2
|\
| * Rename `create_test_json_resource` to `create_test_resource` (#8759)Richard van der Hoff2020-11-161-2/+2
| | | | | | | | | | The root resource isn't necessarily a JsonResource, so rename this method accordingly, and update a couple of test classes to use the method rather than directly manipulating self.resource.
* | use global make_request() directly where we have a custom ResourceRichard van der Hoff2020-11-153-21/+60
|/ | | | | | Where we want to render a request against a specific Resource, call the global make_request() function rather than the one in HomeserverTestCase, allowing us to pass in an appropriate `Site`.
* Fix race for concurrent downloads of remote media. (#8682)Erik Johnston2020-10-301-0/+277
| | | Fixes #6755
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-1/+1
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Don't require hiredis to run unit tests (#8680)Erik Johnston2020-10-291-1/+10
|
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-272-2/+2
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Solidify the HomeServer constructor. (#8515)Jonathan de Jong2020-10-152-3/+3
| | | | | | This implements a more standard API for instantiating a homeserver and moves some of the dependency injection into the test suite. More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.
* Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_inviteRichard van der Hoff2020-10-151-1/+1
|\ | | | | Simplify `_locally_reject_invite`
| * Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-1/+1
| | | | | | | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
* | Add basic tests for sync/pagination with vector clock tokens. (#8488)Erik Johnston2020-10-141-0/+217
|/ | | | | These are tests for #8439
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-091-1/+1
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* Add unit test for event persister sharding (#8433)Erik Johnston2020-10-022-21/+305
|
* Add EventStreamPosition type (#8388)Erik Johnston2020-09-241-3/+9
| | | | | | | | | | | | | | The idea is to remove some of the places we pass around `int`, where it can represent one of two things: 1. the position of an event in the stream; or 2. a token that partitions the stream, used as part of the stream tokens. The valid operations are then: 1. did a position happen before or after a token; 2. get all events that happened before or after a token; and 3. get all events between two tokens. (Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-081-5/+5
|
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-4/+6
|
* Do not propagate typing notifications from shadow-banned users. (#8176)Patrick Cloke2020-08-261-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-3/+3
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-7/+6
|
* Convert push to async/await. (#7948)Patrick Cloke2020-07-271-2/+4
|
* Convert the message handler to async/await. (#7884)Patrick Cloke2020-07-221-32/+44
|
* Add ability to run multiple pusher instances (#7855)Erik Johnston2020-07-161-0/+193
| | | This reuses the same scheme as federation sender sharding
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-153-172/+246
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support handling registration requests across multiple client readers. (#7830)Patrick Cloke2020-07-131-0/+133
|
* Add ability to shard the federation sender (#7798)Erik Johnston2020-07-102-0/+287
|
* Back out MSC2625 implementation (#7761)Brendan Abolivier2020-07-011-16/+3
|
* Merge branch 'develop' into babolivier/mark_unreadBrendan Abolivier2020-06-152-21/+141
|\
| * Discard RDATA from already seen positions. (#7648)Patrick Cloke2020-06-152-21/+141
| |
* | LintBrendan Abolivier2020-06-121-3/+1
| |
* | Test that a mark_unread action updates the right counter when using a slave ↵Brendan Abolivier2020-06-121-0/+15
| | | | | | | | store
* | Fix testsBrendan Abolivier2020-06-101-3/+3
|/
* Performance improvements and refactor of Ratelimiter (#7595)Andrew Morgan2020-06-051-7/+2
| | | | | | | | | | While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both: * Rather undocumented, and * causing a *lot* of config checks This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. Best to be reviewed commit-by-commit.
* Fix recording of federation stream token (#7564)Richard van der Hoff2020-05-261-0/+71
| | | | | | | | A couple of changes of significance: * remove the `_last_ack < federation_position` condition, so that updates will still be correctly processed after restart * Correctly wire up send_federation_ack to the right class.
* fix mypy for tests/replication (#7518)Richard van der Hoff2020-05-182-13/+7
|
* Fix limit logic for AccountDataStream (#7384)Richard van der Hoff2020-05-151-0/+117
| | | | | | Make sure that the AccountDataStream presents complete updates, in the right order. This is much the same fix as #7337 and #7358, but applied to a different stream.
* Clean up replication unit tests. (#7490)Erik Johnston2020-05-137-51/+15
|
* Fix catchup-on-reconnect for the Federation Stream (#7374)Richard van der Hoff2020-05-053-12/+94
| | | | looks like we managed to break this during the refactorathon.
* Thread through instance name to replication client. (#7369)Erik Johnston2020-05-013-6/+6
| | | For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
* Use `stream.current_token()` and remove `stream_positions()` (#7172)Erik Johnston2020-05-014-34/+26
| | | | We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-292-2/+5
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352)Erik Johnston2020-04-291-0/+62
| | | | | | | | | | | | | | For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes #7334.
* Fix limit logic for EventsStream (#7358)Richard van der Hoff2020-04-294-21/+458
| | | | | | | | | | | | | | | | | | | * Factor out functions for injecting events into database I want to add some more flexibility to the tools for injecting events into the database, and I don't want to clutter up HomeserverTestCase with them, so let's factor them out to a new file. * Rework TestReplicationDataHandler This wasn't very easy to work with: the mock wrapping was largely superfluous, and it's useful to be able to inspect the received rows, and clear out the received list. * Fix AssertionErrors being thrown by EventsStream Part of the problem was that there was an off-by-one error in the assertion, but also the limit logic was too simple. Fix it all up and add some tests.
* Add some replication tests (#7278)Erik Johnston2020-04-282-11/+298
| | | | | | | Specifically some tests for the typing stream, which means we test streams that fetch missing updates via HTTP (rather than via the DB). We also shuffle things around a bit so that we create two separate `HomeServer` objects, rather than trying to insert a slaved store into places. Note: `test_typing.py` is heavily inspired by `test_receipts.py`
* Another go at fixing one-word commands (#7326)Richard van der Hoff2020-04-221-0/+42
| | | I messed this up last time I tried (#7239 / e13c6c7).
* Add ability to run replication protocol over redis. (#7040)Erik Johnston2020-04-221-2/+2
| | | This is configured via the `redis` config options.
* Move client command handling out of TCP protocol (#7185)Erik Johnston2020-04-063-29/+25
| | | The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
* Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-252-23/+84
| | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* Convert `*StreamRow` classes to inner classes (#7116)Richard van der Hoff2020-03-231-2/+2
| | | | | This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.
* Read the room version from database when fetching events (#6874)Richard van der Hoff2020-03-041-0/+10
| | | | | This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
* Add a `make_event_from_dict` method (#6858)Richard van der Hoff2020-02-071-4/+8
| | | | | | | ... and use it in places where it's trivial to do so. This will make it easier to pass room versions into the FrozenEvent constructors.
* Port synapse.replication.tcp to async/await (#6666)Erik Johnston2020-01-161-1/+1
| | | | | | | | | | * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-2/+2
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Add database config class (#6513)Erik Johnston2019-12-181-2/+4
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Fixup testsErik Johnston2019-12-061-1/+4
|
* Implementation of MSC2314 (#6176)Amber Brown2019-11-282-0/+7
|
* Use new EventPersistenceStoreErik Johnston2019-10-232-3/+8
|
* Run Black on the tests again (#5170)Amber Brown2019-05-103-26/+14
|
* Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-022-28/+161
| | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
* Move replication.tcp.streams into a packageRichard van der Hoff2019-03-271-1/+1
|
* Fix bug where read-receipts lost their timestamps (#4927)Richard van der Hoff2019-03-254-0/+148
| | | | | Make sure that they are sent correctly over the replication stream. Fixes: #4898
* Add rate-limiting on registration (#4735)Brendan Abolivier2019-03-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test
* Clean up event accesses and testsErik Johnston2018-11-021-2/+2
| | | | | This is in preparation to refactor FrozenEvent to support different event formats for different room versions
* Disable frozen dicts by default (#3987)Amber Brown2018-10-021-1/+5
|
* Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-32/+3
| | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix tests on postgresql (#3740)Amber Brown2018-09-044-124/+116
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-204-67/+64
|\ | | | | | | erikj/refactor_state_handler
| * Run tests under PostgreSQL (#3423)Amber Brown2018-08-131-0/+1
| |
| * Run black.black2018-08-104-67/+63
| |
* | Fix testsErik Johnston2018-08-091-1/+2
|/
* Fix unit testsRichard van der Hoff2018-07-251-7/+30
| | | | | | on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.
* Fix unit testsErik Johnston2018-07-231-3/+5
|
* run isortAmber Brown2018-07-094-13/+15
|
* remove dead filter_events_for_clientsRichard van der Hoff2018-06-121-8/+0
| | | | | | This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.
* s/replication_client/federation_client/Erik Johnston2018-03-131-1/+1
|
* Fix testsErik Johnston2018-03-131-1/+1
|
* Fix unit testsErik Johnston2018-02-201-4/+6
|
* Remove context.push_actionsErik Johnston2018-02-151-1/+4
|
* Store state groups separately from events (#2784)Erik Johnston2018-02-061-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Matthew's fixes to the unit testsRichard van der Hoff2018-01-221-2/+6
| | | | Extracted from https://github.com/matrix-org/synapse/pull/2820
* Remove test of replication resourceErik Johnston2017-04-111-204/+0
|
* Change slave storage to use new replication interfaceErik Johnston2017-04-031-8/+22
| | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication.
* Fix unit testErik Johnston2017-03-201-1/+1
|
* Remote membership tests for replicationErik Johnston2017-01-311-43/+0
| | | | | This is because it now relies of the caches stream, which only works on postgres. We are trying to test with sqlite.
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-29/+0
|
* Derive current_state_events from state groupsErik Johnston2017-01-201-28/+17
|
* Shuffle receipt handler around so that worker apps don't need to load itErik Johnston2016-11-231-1/+1
|
* Reduce DB hits for replicationErik Johnston2016-09-232-3/+3
| | | | | | | | | | | | | | Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly.
* Correctly handle the difference between prev and current stateErik Johnston2016-08-312-9/+5
|
* Replace context.current_state with context.current_state_idsErik Johnston2016-08-251-1/+8
|
* Add `create_requester` functionRichard van der Hoff2016-07-261-9/+11
| | | | | Wrap the `Requester` constructor with a function which provides sensible defaults, and use it throughout
* Remove room name & alias testDavid Baker2016-06-241-41/+0
| | | | as get_room_name_and_alias is now gone
* Move typing handler out of the Handlers objectMark Haines2016-05-171-1/+1
|
* Move the presence handler out of the Handlers objectMark Haines2016-05-161-1/+1
|
* Add a slaved datastore for account dataMark Haines2016-05-131-0/+56
|
* Replicate push actionsMark Haines2016-04-211-0/+43
|
* Merge pull request #738 from matrix-org/markjh/slaved_receiptsMark Haines2016-04-193-3/+43
|\ | | | | Add a slaved receipts store
| * Add a slaved receipts storeMark Haines2016-04-193-3/+43
| |
* | Replicate get_invited_rooms_for_userMark Haines2016-04-191-0/+12
|/
* Add tests for redactionsMark Haines2016-04-072-2/+51
|
* Add sensible __eq__ operators inside the tests.Mark Haines2016-04-071-1/+28
| | | | | Rather than adding them globally. This limits the changes to only affect the tests.
* Add tests for get_latest_event_ids_in_room and get_current_stateMark Haines2016-04-071-0/+62
|
* Test that room membership is replicatedMark Haines2016-04-061-8/+63
|
* Add a slaved events store classMark Haines2016-04-064-0/+199
| | | | | Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store.
* Add a replication stream for state groupsMark Haines2016-03-301-3/+27
|
* Add replication stream for pushersMark Haines2016-03-151-0/+1
|
* Merge branch 'develop' into markjh/pushrule_streamMark Haines2016-03-041-7/+10
|\
| * Pass whole requester to ratelimitingDaniel Wagner-Hall2016-03-031-7/+10
| | | | | | | | This will enable more detailed decisions
* | Hook push rules up to the replication APIMark Haines2016-03-021-2/+4
|/
* Add a /replication API for extracting the updates that happened onMark Haines2016-03-012-0/+193
synapse This is necessary for replicating the data in synapse to be visible to a separate service because presence and typing notifications aren't stored in a database so won't be visible to another process. This API can be used to either get the raw data by requesting the tables themselves or to just receive notifications for updates by following the streams meta-stream. Returns updates for each table requested a JSON array of arrays with a row for each row in the table. Each table is prefixed by a header row with the: name of the table, current stream_id position for the table, number of rows, number of columns and the names of the columns. This is followed by the rows that have been added to the server since the requester last asked. The API has a timeout and is hooked up to the notifier so that a slave can long poll for updates.