summary refs log tree commit diff
path: root/synapse/push/pusherpool.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-101-1/+4
| | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* Use direct references for some configuration variables (part 2) (#10812)Patrick Cloke2021-09-151-1/+1
|
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Add a module type for account validity (#9884)Brendan Abolivier2021-07-161-16/+8
| | | | | This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-1/+1
|
* Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-191-3/+5
| | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Test that we require validated email for email pushers (#9496)Erik Johnston2021-02-251-0/+6
|
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-241-1/+3
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Fix deleting pushers when using sharded pushers. (#9465)Erik Johnston2021-02-221-6/+17
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-5/+4
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix-up assertions about last stream token in push (#9020)Patrick Cloke2021-01-051-5/+5
| | | The last stream token is always known and we do not need to handle none.
* Convert internal pusher dicts to attrs classes. (#8940)Patrick Cloke2020-12-161-65/+70
| | | This improves type hinting and should use less memory.
* Fix handling of stream tokens for push. (#8943)Patrick Cloke2020-12-151-3/+2
| | | | Removes faulty assertions and fixes the logic to ensure the max stream token is always set.
* Add type hints for HTTP and email pushers. (#8880)Patrick Cloke2020-12-071-18/+18
|
* Start fewer opentracing spans (#8640)Erik Johnston2020-10-261-2/+16
| | | | | | | #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.
* Fix not sending events over federation when using sharded event persisters ↵Erik Johnston2020-10-141-2/+8
| | | | | | | | | | | | | | | | | (#8536) * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.
* Don't push if an user account has expired (#8353)Mathieu Velten2020-09-231-0/+18
|
* Clean up `Notifier.on_new_room_event` code path (#8288)Erik Johnston2020-09-101-1/+1
| | | | | | | | | | | | | The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it. This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens. The valid operations here are: 1. Is a position before or after a token 2. Fetching all events between two tokens 3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C. Future PR will change the token type to a dedicated type.
* Fixup pusher pool notifications (#8287)Erik Johnston2020-09-091-3/+16
| | | | | `pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed. I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
* Revert "Fixup pusher pool notifications"Erik Johnston2020-09-091-16/+3
| | | | This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
* Fixup pusher pool notificationsErik Johnston2020-09-091-3/+16
|
* Convert push to async/await. (#7948)Patrick Cloke2020-07-271-42/+28
|
* Add ability to run multiple pusher instances (#7855)Erik Johnston2020-07-161-36/+42
| | | This reuses the same scheme as federation sender sharding
* Refactor getting replication updates from database. (#7636)Erik Johnston2020-06-161-3/+1
| | | The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
* Fix concurrent modification errors in pusher metrics (#7106)Richard van der Hoff2020-03-191-9/+18
| | | add a lock to try to make this metric actually work
* Add prometheus metrics for the number of active pushers (#7103)Richard van der Hoff2020-03-191-1/+23
|
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Clean up startup for the pusher (#6558)Richard van der Hoff2019-12-181-4/+6
| | | | | | | | | | | | | | | | | | | | | | | * Remove redundant python2 support code `str.decode()` doesn't exist on python3, so presumably this code was doing nothing * Filter out pushers with corrupt data When we get a row with unparsable json, drop the row, rather than returning a row with null `data`, which will then cause an explosion later on. * Improve logging when we can't start a pusher Log the ID to help us understand the problem * Make email pusher setup more robust We know we'll have a `data` member, since that comes from the database. What we *don't* know is if that is a dict, and if that has a `brand` member, and if that member is a string.
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-3/+1
| | | * update version of black and also fix the mypy config being overridden
* Replace returnValue with return (#5736)Amber Brown2019-07-231-3/+3
|
* Run Black. (#5482)Amber Brown2019-06-201-44/+53
|
* Add testErik Johnston2019-06-071-7/+23
|
* Make starting pushers faster during start upErik Johnston2019-04-021-4/+24
| | | | | | | | | | | We start all pushers on start up and immediately start a background process to fetch push to send. This makes start up incredibly painful when dealing with many pushers. Instead, let's do a quick fast DB check to see if there *may* be push to send and only start the background processes for those pushers. We also stagger starting up and doing those checks so that we don't try and handle all pushers at once.
* Add missing returnErik Johnston2019-02-221-0/+1
|
* Correctly handle PusherConfigExceptionErik Johnston2019-02-221-0/+9
|
* bail out early in on_new_receipts if no pushers (#4706)Richard van der Hoff2019-02-211-0/+8
|
* Fix start_pushers vs _start_pushers confusionRichard van der Hoff2018-10-241-3/+3
|
* Run PusherPool.start as a background processRichard van der Hoff2018-10-221-5/+7
| | | | | We don't do anything with the result, so this is needed to give this code a logcontext.
* Make on_started synchronous tooRichard van der Hoff2018-10-221-2/+14
| | | | | | This brings it into line with on_new_notifications and on_new_receipts. It requires a little bit of hoop-jumping in EmailPusher to load the throttle params before the first loop.
* Remove redundant run_as_background_process() from pusherpoolRichard van der Hoff2018-10-221-40/+7
| | | | | | | `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.
* move get_all_pushers call downRichard van der Hoff2018-10-221-3/+9
| | | | simplifies the interface to _start_pushers
* Factor PusherPool._start_pusher out of _start_pushersRichard van der Hoff2018-10-221-18/+33
| | | | | ... and use it from start_pusher_by_id. This mostly simplifies start_pusher_by_id.
* Rename _refresh_pusherRichard van der Hoff2018-10-221-2/+3
| | | | | This is public (or at least, called from outside the class), so ought to have a better name.
* Fix logcontexts for running pushersRichard van der Hoff2018-08-171-2/+15
| | | | | | | First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.
* Remove run_on_reactor (#3395)Amber Brown2018-06-141-3/+0
|
* Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-2/+6
|\
| * Add missing consumeErrorsRichard van der Hoff2018-04-271-2/+6
| | | | | | | | | | In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.
* | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-8/+12
|/ | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* Remove pushers when deleting access tokensRichard van der Hoff2017-11-291-9/+15
| | | | | Whenever an access token is invalidated, we should remove the associated pushers.
* Remove preserve_context_over_{fn, deferred}Richard van der Hoff2017-11-141-3/+3
| | | | | Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-3/+3
| | | | what could possibly go wrong
* Only load jinja2 templates onceErik Johnston2017-05-221-3/+4
| | | | | Instead of every time a new email pusher is created, as loading jinja2 templates is slow.
* Preserve some logcontextsErik Johnston2016-08-241-5/+7
|
* Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to ↵Erik Johnston2016-08-151-4/+4
| | | | slaves.
* Unused importDavid Baker2016-05-041-1/+0
|
* Catch all exceptions when creating a pusherDavid Baker2016-05-041-2/+2
|
* Add an email pusher for new usersDavid Baker2016-04-291-0/+1
| | | | If they registered with an email address and email notifs are enabled on the HS
* Optionally split out the pushers into a separate processMark Haines2016-04-211-0/+4
|
* Unused member variableDavid Baker2016-04-121-1/+0
|
* indentingDavid Baker2016-04-071-1/+1
|
* defer.gatherResults rather than doing all the pokes in seriesDavid Baker2016-04-071-2/+16
|
* Send badge count pushes.David Baker2016-04-071-1/+19
| | | | Also fix bugs with retrying.
* Make pushers use the event_push_actions table instead of listening on an ↵David Baker2016-04-061-34/+41
| | | | | | | event stream & running the rules again. Sytest passes, but remaining to do: * Make badges work again * Remove old, unused code
* Make select more sensible when dseleting access tokens, rename pusher ↵David Baker2016-03-111-1/+1
| | | | deletion to match access token deletion and make exception arg optional.
* Fix cache invalidation so deleting access tokens (which we did when changing ↵David Baker2016-03-111-4/+4
| | | | password) actually takes effect without HS restart. Reinstate the code to avoid logging out the session that changed the password, removed in 415c2f05491ce65a4fc34326519754cd1edd9c54
* Fix relative imports so they work in both py3 and py27Mark Haines2016-03-081-1/+1
|
* Remove dead code for setting device specific rules.Mark Haines2016-02-181-28/+20
| | | | | | It wasn't possible to hit the code from the API because of a typo in parsing the request path. Since no-one was using the feature we might as well remove the dead code.
* Fix up logcontextsErik Johnston2016-02-081-4/+5
|
* Merge branch 'develop' into push_badge_countsDavid Baker2016-01-191-13/+13
|\
| * Rename 'user_name' to 'user_id' in push to make it consistent with the rest ↵Mark Haines2016-01-131-15/+15
| | | | | | | | of the code
* | Use the unread notification count to send accurate badge counts in push ↵David Baker2016-01-131-15/+0
|/ | | | notifications.
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Simplify LoginHander and AuthHandlerDaniel Wagner-Hall2015-08-121-7/+4
| | | | | | | | | * Merge LoginHandler -> AuthHandler * Add a bunch of documentation * Improve some naming * Remove unused branches I will start merging the actual logic of the two handlers shortly
* Also log when we've started pushersErik Johnston2015-04-291-0/+2
|
* Only start pushers when synapse has fully startedErik Johnston2015-04-291-3/+0
|
* Start pushers on reactor threadErik Johnston2015-04-291-0/+3
|
* Decode the data json in the storage layer (was moved but this part was missed)David Baker2015-04-291-1/+5
|
* Merge branch 'develop' of github.com:matrix-org/synapse into postgresErik Johnston2015-04-281-15/+61
|\
| * pep8David Baker2015-03-261-2/+4
| |
| * Delete pushers when changing passwordDavid Baker2015-03-261-2/+18
| |
| * pep8 blank linesDavid Baker2015-03-251-1/+0
| |
| * Allow multiple pushers for a single app ID & pushkey, honouring the 'append' ↵David Baker2015-03-251-10/+37
| | | | | | | | flag in the API.
| * 1) Pushers are now associated with an access tokenDavid Baker2015-03-241-4/+5
| | | | | | | | 2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
* | Move encoding and decoding of JSON into storage layerErik Johnston2015-04-161-7/+1
|/
* Use encode_canonical_json for pushesErik Johnston2015-02-111-1/+3
|
* Blunty replace json with simplejsonErik Johnston2015-02-111-1/+1
|
* s/instance_handle/profile_tag/David Baker2015-02-031-6/+6
|
* Code style fixes.Mark Haines2015-01-291-1/+1
|
* Easy on the newlinesDavid Baker2015-01-281-1/+0
|
* 2015David Baker2015-01-281-1/+1
|
* Unused importDavid Baker2015-01-281-1/+0
|
* Reset badge count to zero when last active time is bumpedDavid Baker2015-01-281-0/+17
|
* add instance_handles to pushers so we have a way to refer to them even if ↵David Baker2015-01-201-3/+6
| | | | the push token changes.
* Require device language when adding a pusher.David Baker2015-01-161-3/+5
| | | | | Because this seems like it might be useful to do sooner rather than later.
* Honour the 'rejected' return from push gatewaysDavid Baker2015-01-131-0/+12
| | | | | | | Add a timestamp to push tokens so we know the last time they we got them from the device. Send it to the push gateways so it can determine whether its failure is more recent than the token. Stop and remove pushers that have been rejected.
* after a few rethinks, a working implementation of pushers.David Baker2014-12-181-21/+26
|
* Update to app_id / app_instance_id (partially) and mangle to be PEP8 compliant.David Baker2014-12-031-29/+46
|
* More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP ↵David Baker2014-11-211-2/+6
| | | | pokes work or not yet but the retry semantics are pretty good.
* Start creating a module to do generic notifications (just prints them to ↵David Baker2014-11-191-0/+94
stdout currently!)