summary refs log tree commit diff
path: root/synapse/push/emailpusher.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Encode JSON responses on a thread in C, mk2 (#10905)Erik Johnston2021-09-281-1/+1
| | | | | | | | | | Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library. Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this: 1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or 2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types. I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-3/+3
|
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-1/+8
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-1/+1
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-2/+2
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-11/+15
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix-up assertions about last stream token in push (#9020)Patrick Cloke2021-01-051-6/+1
| | | The last stream token is always known and we do not need to handle none.
* Convert internal pusher dicts to attrs classes. (#8940)Patrick Cloke2020-12-161-13/+14
| | | This improves type hinting and should use less memory.
* Fix handling of stream tokens for push. (#8943)Patrick Cloke2020-12-151-16/+0
| | | | Removes faulty assertions and fixes the logic to ensure the max stream token is always set.
* Add type hints for HTTP and email pushers. (#8880)Patrick Cloke2020-12-071-36/+46
|
* Fix not sending events over federation when using sharded event persisters ↵Erik Johnston2020-10-141-1/+7
| | | | | | | | | | | | | | | | | (#8536) * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.
* Fixup pusher pool notifications (#8287)Erik Johnston2020-09-091-1/+1
| | | | | `pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed. I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
* Revert "Fixup pusher pool notifications"Erik Johnston2020-09-091-1/+1
| | | | This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
* Fixup pusher pool notificationsErik Johnston2020-09-091-1/+1
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-1/+1
|
* Convert sending mail to async/await. (#7557)Erik Johnston2020-05-221-22/+16
| | | | Mainly because sometimes the email push code raises exceptions where the stack traces have gotten lost, which is hopefully fixed by this.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-8/+6
| | | * update version of black and also fix the mypy config being overridden
* Remove redundant returnErik Johnston2019-08-131-1/+0
|
* Handle pusher being deleted during processing.Erik Johnston2019-08-011-6/+13
| | | | | Instead of throwing a StoreError lets break out of processing loop and mark the pusher as stopped.
* Run Black. (#5482)Amber Brown2019-06-201-31/+32
|
* Add testErik Johnston2019-06-071-0/+19
|
* Make starting pushers faster during start upErik Johnston2019-04-021-2/+9
| | | | | | | | | | | We start all pushers on start up and immediately start a background process to fetch push to send. This makes start up incredibly painful when dealing with many pushers. Instead, let's do a quick fast DB check to see if there *may* be push to send and only start the background processes for those pushers. We also stagger starting up and doing those checks so that we don't try and handle all pushers at once.
* Write some tests for the email pusher (#4095)Amber Brown2018-10-301-1/+4
|
* sanity-check the is_processing flagRichard van der Hoff2018-10-241-4/+7
| | | | ... and rename it, for even more sanity
* Make on_started synchronous tooRichard van der Hoff2018-10-221-8/+7
| | | | | | This brings it into line with on_new_notifications and on_new_receipts. It requires a little bit of hoop-jumping in EmailPusher to load the throttle params before the first loop.
* Remove redundant run_as_background_process() from pusherpoolRichard van der Hoff2018-10-221-2/+1
| | | | | | | `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.
* Give pushers their own background logcontextRichard van der Hoff2018-10-221-24/+24
| | | | | | | | Each pusher has its own loop which runs for as long as it has work to do. This should run in its own background thread with its own logcontext, as other similar loops elsewhere in the system do - which means that CPU usage is consistently attributed to that loop, rather than to whatever request happened to start the loop.
* run isortAmber Brown2018-07-091-4/+3
|
* Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-251-2/+2
|
* Improve exception handling for background processesRichard van der Hoff2018-04-271-4/+7
| | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* Only load jinja2 templates onceErik Johnston2017-05-221-12/+3
| | | | | Instead of every time a new email pusher is created, as loading jinja2 templates is slow.
* Ensure we pass positive ints to delay functionErik Johnston2017-02-221-1/+2
|
* Reduce redundant database work in email pusherErik Johnston2016-10-181-0/+4
| | | | | | | Update the last stream ordering if the `get_unread_push_actions_for_user_in_range_for_email` returns no new push actions. This reduces the range that it needs to check next iteration.
* Ignore AlreadyCalled errors on timer cancelErik Johnston2016-07-311-2/+10
|
* Create separate methods for getting messages to pushMark Haines2016-07-281-3/+2
| | | | | | | | | | | for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages.
* Merge remote-tracking branch 'origin/develop' into dbkr/email_unsubscribeDavid Baker2016-06-021-2/+8
|\
| * Disable INCLUDE_ALL_UNREAD_NOTIFSErik Johnston2016-06-021-1/+2
| |
| * if an email pusher specifies a brand param, use itMatthew Hodgson2016-06-021-1/+6
| |
* | Email unsubscribing that may in theory, workDavid Baker2016-06-021-1/+1
|/ | | | Were it not for that fact that you can't use the base handler in the pusher because it pulls in the world. Comitting while I fix that on a different branch.
* Tune email notifs to make them quieter:Matthew Hodgson2016-05-231-8/+18
| | | | | | | | * After initial 10 minute window, only alert every 24h for room notifs * Reset room state after 6h of idleness * Synchronise throttles for messages sent in the same notif, so the 24 hourly notifs 'line up' * Fix the email subjects to say what triggered the notification * Order the rooms in reverse activity order in the email, so the 'reason' room should always come first
* tune email notifs, fix CSS a bit, and add debugging detailsMatthew Hodgson2016-05-161-8/+24
|
* Add constant for throttle multiplierDavid Baker2016-04-291-1/+2
|
* Remove unused functionDavid Baker2016-04-291-4/+0
|
* Remove unused argDavid Baker2016-04-291-2/+2
|
* Remove rather pointless get functionDavid Baker2016-04-291-9/+1
|
* Stop processing notifs once we've sent a mailDavid Baker2016-04-291-0/+1
|
* Put back real delay before mailingDavid Baker2016-04-281-1/+1
|
* Better grammar for multiple messages in a roomDavid Baker2016-04-271-1/+1
| | | | Say who the messages are from if there's no room name, otherwise it's a bit nonsensical
* No inlineCallbacks necessary on thisDavid Baker2016-04-221-1/+0
|
* Flesh out email templatingDavid Baker2016-04-211-0/+7
| | | | Mostly WIP porting the room name calculation logic from the web client so our room names in the email mirror the clients.
* Generate mails from a templateDavid Baker2016-04-201-8/+4
|
* Send a rather basic email notifDavid Baker2016-04-201-9/+23
| | | | Also pep8 fixes
* Add single instance & logging stuffDavid Baker2016-04-191-8/+39
| | | | Copy the stuff over from http pusher that prevents multiple instances of process running at once and sets up logging and measure blocks.
* First bits of emailpusherDavid Baker2016-04-191-0/+214
Mostly logic of when to send an email