summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge tag 'v1.2.1' into shhs shhs-v1.2.1Amber Brown2019-07-261-0/+31
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.2.1 (2019-07-26) ========================== Security update --------------- This release includes *four* security fixes: - Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. ([\#5767](https://github.com/matrix-org/synapse/issues/5767)) - Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to `@lrizika:matrix.org` for identifying and responsibly disclosing this issue. ([0f2ecb961](https://github.com/matrix-org/synapse/commit/0f2ecb961)) - Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to @Dylanger for identifying and responsibly disclosing this issue. ([\#5744](https://github.com/matrix-org/synapse/issues/5744)) - Fix a vulnerability where a federated server could spoof read-receipts from users on other servers. Thanks to @Dylanger for identifying this issue too. ([\#5743](https://github.com/matrix-org/synapse/issues/5743)) Additionally, the following fix was in Synapse **1.2.0**, but was not correctly identified during the original release: - It was possible for a room moderator to send a redaction for an `m.room.create` event, which would downgrade the room to version 1. Thanks to `/dev/ponies` for identifying and responsibly disclosing this issue! ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
| * Fix DoS when there is a cycle in redaction eventsRichard van der Hoff2019-07-261-0/+4
| | | | | | | | | | Make sure that synapse doesn't explode when a redaction redacts itself, or there is a larger cycle.
| * log when a redaction attempts to redact an event in a different roomRichard van der Hoff2019-07-251-0/+27
| |
* | Merge tag 'v1.2.0' into shhs shhs-v1.2Amber H. Brown2019-07-267-254/+304
|\| | | | | | | No changes since v1.2.0rc2.
| * Ignore redactions of m.room.create events (#5701)Richard van der Hoff2019-07-171-0/+12
| |
| * More refactoring in `get_events_as_list` (#5707)Richard van der Hoff2019-07-171-27/+37
| | | | | | | | | | | | | | | | We can now use `_get_events_from_cache_or_db` rather than going right back to the database, which means that (a) we can benefit from caching, and (b) it opens the way forward to more extensive checks on the original event. We now always require the original event to exist before we will serve up a redaction.
| * Fix redaction authentication (#5700)Richard van der Hoff2019-07-171-85/+129
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ensures that redactions are correctly authenticated for recent room versions. There are a few things going on here: * `_fetch_event_rows` is updated to return a dict rather than a list of rows. * Rather than returning multiple copies of an event which was redacted multiple times, it returns the redactions as a list within the dict. * It also returns the actual rejection reason, rather than merely the fact that it was rejected, so that we don't have to query the table again in `_get_event_from_row`. * The redaction handling is factored out of `_get_event_from_row`, and now checks if any of the redactions are valid.
| * Refactor `get_events_as_list` (#5699)Richard van der Hoff2019-07-171-48/+74
| | | | | | | | | | | | | | | | | | | | | | A couple of changes here: * get rid of a redundant `allow_rejected` condition - we should already have filtered out any rejected events before we get to that point in the code, and the redundancy is confusing. Instead, let's stick in an assertion just to make double-sure we aren't leaking rejected events by mistake. * factor out a `_get_events_from_cache_or_db` method, which is going to be important for a forthcoming fix to redactions.
| * Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_dataErik Johnston2019-07-152-6/+28
| |\ | | | | | | Add basic function to get all data for a user out of synapse
| | * Fixup docstringsErik Johnston2019-07-121-7/+5
| | |
| | * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-07-056-39/+25
| | |\ | | | | | | | | | | | | erikj/admin_exfiltrate_data
| | * | Assume key existence. Update docstringsErik Johnston2019-07-051-6/+10
| | | |
| | * | Add basic function to get all data for a user out of synapseErik Johnston2019-07-021-0/+20
| | | |
| * | | Implement access token expiry (#5660)Richard van der Hoff2019-07-122-5/+32
| | | | | | | | | | | | | | | | Record how long an access token is valid for, and raise a soft-logout once it expires.
| * | | fix typo: backgroud -> backgroundAndrew Morgan2019-07-121-4/+4
| | | |
| * | | small typo fix (#5655)Andrew Morgan2019-07-111-1/+1
| | | |
| * | | Remove access-token support from RegistrationStore.register (#5642)Richard van der Hoff2019-07-101-20/+4
| | | | | | | | | | | | | | | | | | | | The 'token' param is no longer used anywhere except the tests, so let's kill that off too.
| * | | Include the original event in /relations (#5626)Andrew Morgan2019-07-091-1/+1
| | | | | | | | | | | | When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.
| * | | Remove support for invite_3pid_guest. (#5625)Richard van der Hoff2019-07-051-47/+0
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | This has never been documented, and I'm not sure it's ever been used outside sytest. It's quite a lot of poorly-maintained code, so I'd like to get rid of it. For now I haven't removed the database table; I suggest we leave that for a future clearout.
| * | remove dead transaction persist code (#5622)Richard van der Hoff2019-07-051-28/+0
| | | | | | | | | | | | this hasn't done anything for years
* | | Merge remote-tracking branch 'origin/develop' into shhs shhs-v1.1.1Amber H. Brown2019-07-054-7/+7
|\| |
| * | Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-044-7/+7
| | |
* | | lintingAmber H. Brown2019-07-051-25/+0
| | |
* | | lintingAmber H. Brown2019-07-051-1/+3
| | |
* | | Merge remote-tracking branch 'origin/develop' into shhsAmber H. Brown2019-07-032-4/+18
|\| |
| * | Fix a number of "Starting txn from sentinel context" warnings (#5605)Richard van der Hoff2019-07-032-4/+18
| |/ | | | | | | Fixes #5602, #5603
* | Merge remote-tracking branch 'origin/develop' into shhsAmber H. Brown2019-07-0233-293/+369
|\|
| * Review commentsErik Johnston2019-06-271-3/+1
| |
| * Use batch_iter and correct docstringErik Johnston2019-06-261-15/+12
| |
| * Rename get_users_whose_devices_changedErik Johnston2019-06-261-3/+3
| |
| * Refactor get_user_ids_changed to pull less from DBErik Johnston2019-06-261-10/+41
| | | | | | | | | | | | | | | | | | | | | | When a client asks for users whose devices have changed since a token we used to pull *all* users from the database since the token, which could easily be thousands of rows for old tokens. This PR changes this to only check for changes for users the client is actually interested in. Fixes #5553
| * Merge pull request #5499 from matrix-org/rav/cleanup_metricsRichard van der Hoff2019-06-241-22/+13
| |\ | | | | | | Cleanups and sanity-checking in cpu and db metrics
| | * blackRichard van der Hoff2019-06-241-6/+2
| | |
| | * Merge branch 'develop' into rav/cleanup_metricsRichard van der Hoff2019-06-2433-254/+306
| | |\
| | * | Simplify PerformanceCounters.update interfaceRichard van der Hoff2019-06-191-11/+12
| | | | | | | | | | | | | | | | | | | | we already have the duration for the update, so may as well use it rather than passing extra params around and recalculating it.
| | * | Remove unused _get_event_countersRichard van der Hoff2019-06-191-7/+1
| | | | | | | | | | | | | | | | This has been redundant since cdb3757942fefdcdc3d33b9c6d7c9e44decefd6f.
| * | | Merge pull request #5498 from matrix-org/rav/fix_clock_reversalRichard van der Hoff2019-06-241-6/+14
| |\ \ \ | | |_|/ | |/| | Use monotonic clock where possible for metrics
| | * | Use monotonic clock where possible for metricsRichard van der Hoff2019-06-191-6/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes intermittent errors observed on Apple hardware which were caused by time.clock() appearing to go backwards when called from different threads. Also fixes a bug where database activity times were logged as 1/1000 of their correct ratio due to confusion between milliseconds and seconds.
| * | | Fix "Unexpected entry in 'full_schemas'" log warning (#5509)Richard van der Hoff2019-06-211-1/+1
| | | | | | | | | | | | | | | | | | | | There is a README.txt which always sets off this warning, which is a bit alarming when you first start synapse. I don't think we need to warn about this.
| * | | BlackErik Johnston2019-06-201-2/+2
| | | |
| * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-06-2034-276/+286
| |\ \ \ | | | | | | | | | | | | | | | erikj/histogram_extremities
| | * | | Run Black. (#5482)Amber Brown2019-06-2033-253/+252
| | | | |
| | * | | Merge pull request #5480 from matrix-org/erikj/extremities_dummy_eventsErik Johnston2019-06-191-0/+29
| | |\ \ \ | | | | | | | | | | | | Add experimental option to reduce extremities.
| | | * | | Add experimental option to reduce extremities.Erik Johnston2019-06-181-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.
| | * | | | Fix typo in deactivation background jobBrendan Abolivier2019-06-191-1/+1
| | | |_|/ | | |/| |
| * | | | Add descriptions and remove redundant set(..)Erik Johnston2019-06-191-3/+5
| | | | |
| * | | | Add metrics for len of new extremities persisted.Erik Johnston2019-06-171-1/+23
| | |/ / | |/| | | | | | | | | | | | | | Of new events being persisted add metrics for total size of forward extremities and number of unchanged, "stale" extremities.
* | | | Merge remote-tracking branch 'origin/develop' into shhsAmber H. Brown2019-06-1823-216/+3337
|\ \ \ \ | | |/ / | |/| |
| * | | Remove Postgres 9.4 support (#5448)Amber Brown2019-06-182-24/+6
| |/ /
| * | Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitiveErik Johnston2019-06-141-1/+2
| |\ \ | | | | | | | | Prometheus histograms are cumalative
| | * | Prometheus histograms are cumalativeErik Johnston2019-06-141-1/+2
| | | |
| * | | Fix background job for deactivated flagBrendan Abolivier2019-06-141-2/+4
| | | |
| * | | Don't send renewal emails to deactivated usersBrendan Abolivier2019-06-142-2/+16
| |/ /
| * | Track deactivated accounts in the database (#5378)Brendan Abolivier2019-06-142-0/+133
| | |
| * | Expose statistics on extrems to prometheus (#5384)Amber Brown2019-06-131-13/+31
| | |
| * | Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into developErik Johnston2019-06-112-1/+5
| |\|
| | * Fix defaults on checking threepidsErik Johnston2019-06-101-1/+1
| | |
| | * Fix bug sending federation transactions with lots of EDUsErik Johnston2019-06-101-0/+4
| | | | | | | | | | | | | | | | | | If we try and send a transaction with lots of EDUs and we run out of space, we call get_new_device_msgs_for_remote with a limit of 0, which then failed.
| * | add monthly active users to phonehome stats (#5252)Neil Johnson2019-06-101-15/+29
| |/ | | | | | | * add monthly active users to phonehome stats
| * Fix key verification when key stored with null valid_until_msErik Johnston2019-06-101-0/+8
| | | | | | | | | | | | | | | | | | | | Some keys are stored in the synapse database with a null valid_until_ms which caused an exception to be thrown when using that key. We fix this by treating nulls as zeroes, i.e. they keys will match verification requests with a minimum_valid_until_ms of zero (i.e. don't validate ts) but will not match requests with a non-zero minimum_valid_until_ms. Fixes #5391.
| * Prevent multiple device list updates from breaking a batch send (#5156)Andrew Morgan2019-06-061-29/+123
| | | | | | fixes #5153
| * Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1Erik Johnston2019-06-065-2/+2352
| |\ | | | | | | Make a full SQL schema
| | * fix maybeAmber H. Brown2019-06-071-20/+2
| | |
| | * just user dir?Amber H. Brown2019-06-071-1/+19
| | |
| | * remove background updates that arent neededAmber H. Brown2019-06-071-33/+1
| | |
| | * add more commentsAmber Brown2019-06-061-0/+5
| | |
| | * add stuff in bg updatesAmber Brown2019-06-041-10/+62
| | |
| | * WHY IS THIS CALLED A SLIGHTLY DIFFERENT THINGAmber Brown2019-06-031-0/+1
| | |
| | * more fixAmber Brown2019-06-031-0/+38
| | |
| | * fix schemasAmber Brown2019-06-034-61/+20
| | |
| | * full schemaAmber Brown2019-06-033-0/+2315
| | |
| | * prepareAmber Brown2019-06-031-2/+14
| | |
| * | Add ability to perform password reset via email without trusting the ↵Andrew Morgan2019-06-064-5/+324
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | identity server (#5377) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368
| * | Merge pull request #5221 from matrix-org/erikj/fix_worker_sytestErik Johnston2019-06-061-1/+11
| |\ \ | | | | | | | | Fix get_max_topological_token to never return None
| | * | Better wordsErik Johnston2019-06-061-1/+1
| | | |
| | * | Fix get_max_topological_token to never return NoneErik Johnston2019-05-211-1/+11
| | | |
| * | | Fix background updates to handle redactions/rejections (#5352)Erik Johnston2019-06-061-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).
| * | | FixErik Johnston2019-06-031-1/+2
| | | |
| * | | Ignore room state with null bytes in for room statsErik Johnston2019-06-031-0/+15
| | |/ | |/|
| * | Merge pull request #5276 from matrix-org/babolivier/account_validity_job_deltaErik Johnston2019-05-311-2/+20
| |\ \ | | | | | | | | Allow configuring a range for the account validity startup job
| | * | Move delta from +10% to -10%Brendan Abolivier2019-05-311-2/+2
| | | |
| | * | Make max_delta equal to period * 10%Brendan Abolivier2019-05-311-4/+3
| | | |
| | * | Allow configuring a range for the account validity startup jobBrendan Abolivier2019-05-281-2/+21
| | | | | | | | | | | | | | | | | | | | When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal. In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
| * | | Merge pull request #5294 from matrix-org/erikj/speed_up_room_statsErik Johnston2019-05-314-87/+111
| |\ \ \ | | | | | | | | | | Speed up room stats background update
| | * | | Add indices. Remove room_ids accidentally addedErik Johnston2019-05-312-13/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have to do this by re-inserting a background update and recreating tables, as the tables only get created during a background update and will later be deleted. We also make sure that we remove any entries that should have been removed but weren't due to a race that has been fixed in a previous commit.
| | * | | Join against events to use its room_id indexErik Johnston2019-05-301-1/+6
| | | | |
| | * | | Move deletion from table inside txnErik Johnston2019-05-301-2/+5
| | | | |
| | * | | Fetch membership counts all at onceErik Johnston2019-05-302-38/+18
| | | | |
| | * | | Get events all at onceErik Johnston2019-05-301-33/+26
| | | | |
| * | | | Add index to temp tableErik Johnston2019-05-301-0/+1
| |/ / /
| * | | Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremitiesErik Johnston2019-05-305-181/+436
| |\ \ \ | | | | | | | | | | Add DB bg update to cleanup extremities.
| | * | | Update synapse/storage/events_bg_updates.pyErik Johnston2019-05-301-1/+1
| | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * | | Rename constantErik Johnston2019-05-301-3/+3
| | | | |
| | * | | Move event background updates to a separate fileErik Johnston2019-05-303-369/+405
| | | | |
| | * | | Fixup comments and loggingErik Johnston2019-05-302-9/+15
| | | | |
| | * | | Log actual number of entries deletedErik Johnston2019-05-292-5/+13
| | | | |
| | * | | Add DB bg update to cleanup extremities.Erik Johnston2019-05-292-0/+205
| | | | | | | | | | | | | | | | | | | | | | | | | Due to #5269 we may have extremities in our DB that we shouldn't have, so lets add a cleanup task such to remove those.
| * | | | Implement the SHHS complexity API (#5216)Amber Brown2019-05-301-1/+49
| |/ / /
| * | | Correctly filter out extremities with soft failed prevs (#5274)Erik Johnston2019-05-291-3/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we receive a soft failed event we, correctly, *do not* update the forward extremity table with the event. However, if we later receive an event that references the soft failed event we then need to remove the soft failed events prev events from the forward extremities table, otherwise we just build up forward extremities. Fixes #5269
| * | | Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schemaBrendan Abolivier2019-05-281-0/+3
| |\| | | | | | | | | | Fix schema update for account validity
| | * | Fix schema update for account validityBrendan Abolivier2019-05-281-0/+3
| | | |
| * | | Fix logging for room stats background updateTravis Ralston2019-05-251-1/+1
| | | |
| * | | Store key validity time in the storage layerRichard van der Hoff2019-05-232-10/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.
| * | | Rewrite store_server_verify_key to store several keys at once (#5234)Richard van der Hoff2019-05-231-26/+39
| | | | | | | | | | | | | | | | | | | | Storing server keys hammered the database a bit. This replaces the implementation which stored a single key, with one which can do many updates at once.
| * | | Simplifications and comments in do_auth (#5227)Richard van der Hoff2019-05-231-1/+1
| | | | | | | | | | | | | | | | | | | | I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.
* | | | Merge remote-tracking branch 'origin/master' into shhsAmber Brown2019-06-045-184/+516
|\ \ \ \
| * | | | Add index to temp tableErik Johnston2019-05-301-0/+1
| | | | |
| * | | | Update synapse/storage/events_bg_updates.pyErik Johnston2019-05-301-1/+1
| | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * | | | Rename constantErik Johnston2019-05-301-3/+3
| | | | |
| * | | | Move event background updates to a separate fileErik Johnston2019-05-303-369/+405
| | | | |
| * | | | Fixup comments and loggingErik Johnston2019-05-302-9/+15
| | | | |
| * | | | Log actual number of entries deletedErik Johnston2019-05-302-5/+13
| | | | |
| * | | | Add DB bg update to cleanup extremities.Erik Johnston2019-05-302-0/+205
| | | | | | | | | | | | | | | | | | | | | | | | | Due to #5269 we may have extremities in our DB that we shouldn't have, so lets add a cleanup task such to remove those.
| * | | | Correctly filter out extremities with soft failed prevs (#5274)Erik Johnston2019-05-301-3/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we receive a soft failed event we, correctly, *do not* update the forward extremity table with the event. However, if we later receive an event that references the soft failed event we then need to remove the soft failed events prev events from the forward extremities table, otherwise we just build up forward extremities. Fixes #5269
* | | | | Merge remote-tracking branch 'origin/develop' into HEADAmber Brown2019-05-2110-27/+703
|\ \ \ \ \ | | |/ / / | |/| | |
| * | | | Room Statistics (#4338)Amber Brown2019-05-216-3/+597
| |/ / /
| * | | Merge pull request #5203 from matrix-org/erikj/aggregate_by_senderErik Johnston2019-05-211-1/+1
| |\ \ \ | | | | | | | | | | Only count aggregations from distinct senders
| | * | | Only count aggregations from distinct sendersErik Johnston2019-05-171-1/+1
| | | | | | | | | | | | | | | | | | | | As a user isn't allowed to send a single emoji more than once.
| * | | | Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactionsErik Johnston2019-05-211-3/+45
| |\ \ \ \ | | | | | | | | | | | | Block attempts to annotate the same event twice
| | * | | | Block attempts to annotate the same event twiceErik Johnston2019-05-201-3/+45
| | | |_|/ | | |/| |
| * | | | Exclude soft-failed events from fwd-extremity candidates. (#5146)Richard van der Hoff2019-05-211-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.
| * | | | Merge pull request #5204 from ↵Brendan Abolivier2019-05-212-12/+62
| |\ \ \ \ | | |/ / / | |/| | / | | | |/ | | |/| matrix-org/babolivier/account_validity_expiration_date Add startup background job for account validity
| | * | Do the select and insert in a single transactionBrendan Abolivier2019-05-211-10/+6
| | | |
| | * | Add startup background job for account validityBrendan Abolivier2019-05-172-12/+66
| | | | | | | | | | | | | | | | If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
* | | | SHHS - Room Join Complexity (#5072)Amber Brown2019-05-201-0/+44
|/ / /
* | | Rename relation types to match MSCErik Johnston2019-05-201-4/+4
| | |
* | | Merge pull request #5209 from matrix-org/erikj/reactions_baseErik Johnston2019-05-204-4/+476
|\ \ \ | | | | | | | | Land basic reaction and edit support.
| * | | Correctly update aggregation counts after redactionErik Johnston2019-05-202-0/+20
| | |/ | |/|
| * | Fixup bsaed on review commentsErik Johnston2019-05-171-15/+17
| | |
| * | Add basic editing supportErik Johnston2019-05-161-2/+58
| | |
| * | Move parsing of tokens out of storage layerErik Johnston2019-05-161-14/+2
| | |
| * | Indirect tuple conversionErik Johnston2019-05-161-0/+6
| | |
| * | Fix relations in worker modeErik Johnston2019-05-162-5/+11
| | |
| * | Add cache to relationsErik Johnston2019-05-161-0/+8
| | |
| * | Add aggregations APIErik Johnston2019-05-161-6/+219
| | |
| * | Actually check for None rather falseyErik Johnston2019-05-151-2/+2
| | |
| * | Add simple pagination APIErik Johnston2019-05-151-0/+80
| | |
| * | Add simple send_relation API and track in DBErik Johnston2019-05-154-0/+93
| | |
* | | Spelling and clarificationsErik Johnston2019-05-171-5/+9
| | |
* | | Update docstringErik Johnston2019-05-161-1/+12
|/ /
* / Make generating SQL bounds for pagination genericErik Johnston2019-05-151-61/+118
|/ | | | | This will allow us to reuse the same structure when we paginate e.g. relations
* Update docstring with correct typeErik Johnston2019-05-151-1/+1
| | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Expose DataStore._get_events as get_events_as_listErik Johnston2019-05-145-29/+53
| | | | This is in preparation for reaction work which requires it.
* Limit the number of EDUs in transactions to 100 as expected by receiver (#5138)Quentin Dufour2019-05-091-1/+1
| | | Fixes #3951.
* Fix infinite loop in presence handlerRichard van der Hoff2019-04-261-0/+18
| | | | Fixes #5102
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-04-1769-3786/+2805
|\ | | | | | | babolivier/account_expiration
| * Merge pull request #5047 from matrix-org/babolivier/account_expirationBrendan Abolivier2019-04-172-18/+159
| |\ | | | | | | Send out emails with links to extend an account's validity period
| * \ Merge pull request #5033 from matrix-org/erikj/fix_schema_deltaErik Johnston2019-04-101-1/+3
| |\ \ | | | | | | | | Fix schema upgrade when dropping tables
| | * | Fix schema upgrade when dropping tablesErik Johnston2019-04-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We need to drop tables in the correct order due to foreign table constraints (on `application_services`), otherwise the DROP TABLE command will fail. Introduced in #4992.
| * | | Merge pull request #5027 from matrix-org/babolivier/account_expirationBrendan Abolivier2019-04-092-0/+54
| |\ \ \ | | |/ / | |/| | Add time-based account expiration
| * | | Rewrite Datastore.get_server_verify_keysRichard van der Hoff2019-04-091-28/+46
| | | | | | | | | | | | | | | | Rewrite this so that it doesn't hammer the database.
| * | | Remove unused server_tls_certificates functions (#5028)Richard van der Hoff2019-04-083-59/+5
| | | | | | | | | | | | | | | | These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.
| * | | drop tables listed in #1830 (#4992)Neil Johnson2019-04-0824-752/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities
| * | | Merge pull request #5001 from matrix-org/rav/keyring_cleanupsRichard van der Hoff2019-04-081-2/+2
| |\ \ \ | | | | | | | | | | Cleanups in the Keyring
| | * | | Fix docstring on get_server_keys_jsonRichard van der Hoff2019-04-041-2/+2
| | | | |
| * | | | Clean up the database pagination code (#5007)Amber Brown2019-04-052-67/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * rewrite & simplify * changelog * cleanup potential sql injection
| * | | | Add functions to delete a groupErik Johnston2019-04-031-0/+37
| | | | |
| * | | | Fix grammar and document get_current_users_in_room (#4998)Andrew Morgan2019-04-031-1/+1
| |/ / /
| * | | Remove presence lists (#4989)Neil Johnson2019-04-034-96/+20
| | | | | | | | | | | | Remove presence list support as per MSC 1819
| * | | Merge pull request #4982 from matrix-org/erikj/msc1915Erik Johnston2019-04-032-0/+141
| |\ \ \ | | | | | | | | | | Implement MSC1915 - 3PID unbind APIs
| | * | | Fixup docstringsErik Johnston2019-04-022-5/+5
| | | | |
| | * | | Grandfather in existing user threepidsErik Johnston2019-04-012-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assume, as we did before, that users bound their threepid to one of the trusted identity servers. So we simply fill the new table with all threepids in `user_threepids` joined with the trusted identity servers.
| | * | | Track IS used to bind 3PIDsErik Johnston2019-04-012-0/+104
| | | | | | | | | | | | | | | | | | | | | | | | | This will then be used to know which IS to default to when unbinding the threepid.
| * | | | Run black on the rest of the storage module (#4996)Amber Brown2019-04-0341-2455/+2130
| | | | |
| * | | | Make starting pushers faster during start upErik Johnston2019-04-021-0/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We start all pushers on start up and immediately start a background process to fetch push to send. This makes start up incredibly painful when dealing with many pushers. Instead, let's do a quick fast DB check to see if there *may* be push to send and only start the background processes for those pushers. We also stagger starting up and doing those checks so that we don't try and handle all pushers at once.
| * | | | Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-022-11/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
| * | | | Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-011-1/+2
| |/ / / | | | | | | | | | | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
| * | | Merge pull request #4955 from matrix-org/rav/merge_state_into_eventsRichard van der Hoff2019-03-281-3/+0
| |\ \ \ | | | | | | | | | | Combine the CurrentStateDeltaStream into the EventStream
| | * | | Combine the CurrentStateDeltaStream into the EventStreamRichard van der Hoff2019-03-271-3/+0
| | | | |
| * | | | Run `black` on some storage modules that the stats branch touches (#4959)Amber Brown2019-03-292-380/+335
| |/ / /
* | | / Add management endpoints for account validityBrendan Abolivier2019-04-171-12/+17
| |_|/ |/| |
* | | Send out emails with links to extend an account's validity periodBrendan Abolivier2019-04-172-18/+159
| |/ |/|
* | Add account expiration featureBrendan Abolivier2019-04-093-1/+55
|/
* Use the state event amount for userdir import batching, not room count (#4944)Amber Brown2019-03-271-8/+20
|
* Fix bug where read-receipts lost their timestamps (#4927)Richard van der Hoff2019-03-251-1/+3
| | | | | Make sure that they are sent correctly over the replication stream. Fixes: #4898
* Refactor out state delta handling into its own class (#4917)Amber Brown2019-03-252-64/+76
|
* Fix upsertErik Johnston2019-03-211-0/+1
|
* comment block_roomErik Johnston2019-03-201-0/+9
|
* Allow blocking a room multiple timesErik Johnston2019-03-201-2/+4
|
* Batching in the user directory import (#4900)Amber Brown2019-03-211-0/+13
|
* Fix user directory background update (#4887)Erik Johnston2019-03-191-1/+1
|
* Migrate the user directory initial population to a background task (#4864)Amber Brown2019-03-193-96/+312
|
* Merge pull request #4846 from matrix-org/hawkowl/userdir-searchErik Johnston2019-03-144-75/+92
|\ | | | | Improve searching in the userdir
| * fixupAmber Brown2019-03-131-1/+1
| |
| * fixupAmber Brown2019-03-132-55/+11
| |
| * fixupAmber Brown2019-03-121-1/+1
| |
| * fixupAmber Brown2019-03-121-1/+2
| |
| * fixupAmber Brown2019-03-122-113/+69
| |
| * fixupAmber Brown2019-03-121-1/+1
| |
| * use the old methodAmber Brown2019-03-121-3/+2
| |
| * fixAmber Brown2019-03-121-1/+1
| |
| * pep8Amber Brown2019-03-121-2/+1
| |
| * fixesAmber Brown2019-03-122-9/+56
| |
| * Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-searchAmber Brown2019-03-112-15/+2
| |\
| * | initialAmber Brown2019-03-113-3/+62
| | |
* | | Transfer local user's push rules on room upgrade (#4838)Andrew Morgan2019-03-121-0/+57
| |/ |/| | | Transfer push rules (notifications) on room upgrade
* | Add comment to schemaErik Johnston2019-03-081-0/+2
| |
* | Make `prev_state` field optionalErik Johnston2019-03-081-15/+0
|/ | | | | | | The `prev_state` field on events is not specced and so synapse shouldn't explode if an event is missing the field. Fixes #4787
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-03-083-181/+156
|\ | | | | | | erikj/soft_fail_impl
| * Rewrite userdir to be faster (#4537)Amber Brown2019-03-072-181/+137
| |
| * Add docstrings from matrix-org-hotfixesErik Johnston2019-03-061-0/+19
| |
* | Implement soft failErik Johnston2019-03-061-0/+1
|/
* Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_roomErik Johnston2019-03-051-0/+22
|\ | | | | Stop backpaginating when events not visible
| * s/get_forward_events/get_successor_events/Erik Johnston2019-03-041-2/+2
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-03-044-43/+120
| |\ | | | | | | | | | erikj/stop_fed_not_in_room
| * | Stop backpaginating when events not visibleErik Johnston2019-02-201-0/+22
| | |
* | | Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keysErik Johnston2019-03-053-541/+546
|\ \ \ | | | | | | | | Allow /keys/{changes,query} API to run on worker
| * | | Fixup slave storesErik Johnston2019-03-043-541/+546
| | |/ | |/|
* / | Add some debug about processing read receipts.Richard van der Hoff2019-03-041-6/+20
|/ / | | | | | | | | I'm hoping to establish which rooms are having lots of RRs sent for them, and how old the events are when they are sent.
* | Move /account/3pid to client_readerErik Johnston2019-02-271-33/+33
| |
* | Limit cache invalidation replication line length (#4748)Erik Johnston2019-02-271-4/+11
| |
* | Add database version to phonehome stats. (#4753)Richard van der Hoff2019-02-273-0/+42
| |
* | Fix state cache invalidation on workersErik Johnston2019-02-221-6/+34
|/
* UPSERT many functionality (#4644)Amber Brown2019-02-201-12/+134
|
* Merge pull request #4671 from matrix-org/erikj/state_cache_invalidationErik Johnston2019-02-192-25/+58
|\ | | | | Batch cache invalidation over replication
| * DocsErik Johnston2019-02-191-2/+3
| |
| * Use itertoolsErik Johnston2019-02-191-2/+2
| |
| * DocumentationErik Johnston2019-02-191-4/+4
| |
| * Batch cache invalidation over replicationErik Johnston2019-02-182-25/+57
| | | | | | | | | | | | | | | | | | | | Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.
* | Split /login into client_readerErik Johnston2019-02-181-41/+41
|/
* Split out registration to workerErik Johnston2019-02-181-115/+115
| | | | | | | | This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.
* Add configurable room list publishing rulesErik Johnston2019-02-141-0/+25
| | | | | This allows specifying who and what is allowed to be published onto the public room list
* Run `black` on user directory code (#4635)Amber Brown2019-02-131-87/+66
|
* Merge pull request #4627 from matrix-org/erikj/user_ips_analyzeErik Johnston2019-02-122-3/+31
|\ | | | | Analyze user_ips before running deduplication
| * Fix pep8Erik Johnston2019-02-121-1/+1
| |
| * Analyze user_ips before running deduplicationErik Johnston2019-02-122-3/+31
| | | | | | | | | | | | | | Due to the table locks taken out by the naive upsert, the table statistics may be out of date. During deduplication it is important that the correct index is used as otherwise a full table scan may be incorrectly used, which can end up thrashing the database badly.
* | Reduce user_ips bloat during dedupe background updateErik Johnston2019-02-121-3/+60
|/ | | | | | | | | | | | The background update to remove duplicate rows naively deleted and reinserted the duplicates. For large tables with a large number of duplicates this causes a lot of bloat (with postgres), as the inserted rows are appended to the table, since deleted rows will not be overwritten until a VACUUM has happened. This should hopefully also help ensure that the query in the last batch uses the correct index, as inserting a large number of new rows without analyzing will upset the query planner.
* Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updatingErik Johnston2019-02-111-0/+21
|\ | | | | add updating of backup versions
| * add new endpoint to update backup versionsHubert Chathi2019-02-061-0/+21
| |
* | Fix 'no unique or exclusion constraint' error (#4591)Richard van der Hoff2019-02-081-6/+21
| | | | | | | | | | Add more tables to the list of tables which need a background update to complete before we can upsert into them, which fixes a race against the background updates.
* | Filter user directory state query to a subset of state events (#4462)Andrew Morgan2019-02-051-4/+12
|/ | | | | | * Filter user directory state query to a subset of state events * Add changelog
* Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatableAndrew Morgan2019-01-311-11/+27
|\ | | | | Copy over non-federatable trait on room upgrade
| * Raise an exception instead of returning NoneAndrew Morgan2019-01-311-2/+9
| |
| * New function for getting room's create eventAndrew Morgan2019-01-311-11/+20
| |
* | Fix infinite loop when an event is redacted in a v3 room (#4535)Richard van der Hoff2019-01-311-5/+32
|/
* Fix flake8 (#4519)Amber Brown2019-01-303-86/+86
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-01-291-0/+23
|\ | | | | | | erikj/redactions_eiah
| * Refactor event building into EventBuilderErik Johnston2019-01-291-0/+23
| | | | | | | | | | This is so that everything is done in one place, making it easier to change the event format based on room version
* | Update synapse/storage/events_worker.pyErik Johnston2019-01-291-1/+1
| |
* | Check redaction state when event is pulled out of the databaseErik Johnston2019-01-291-0/+13
| |
* | Fix typoErik Johnston2019-01-291-1/+1
| |
* | Remove unused argErik Johnston2019-01-291-2/+1
| |
* | Update synapse/storage/events_worker.pyRichard van der Hoff2019-01-291-1/+1
| | | | | | Co-Authored-By: erikjohnston <erikj@jki.re>
* | Implement rechecking of redactionsErik Johnston2019-01-291-1/+25
|/
* Fix typo in upserts code (#4505)Amber Brown2019-01-291-1/+1
| | | | | | * fix obvious problem :| * changelog
* Fix UPSERTs on SQLite 3.24+ (#4477)Amber Brown2019-01-283-12/+20
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Andrew Morgan2019-01-282-10/+9
|\ | | | | | | anoa/dm_room_upgrade
| * Merge pull request #4470 from matrix-org/erikj/require_format_versionErik Johnston2019-01-251-6/+2
| |\ | | | | | | Require event format version to parse or create events
| | * Replace missed usages of FrozenEventErik Johnston2019-01-251-6/+2
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-01-252-2/+25
| |\ \ | | | | | | | | | | | | erikj/sqlite_native_upsert
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-11 16:32:46 +0000