| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.
The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.
`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".
See MSC3771 and MSC3773 for where this data will be useful.
|
| |
|
|
|
|
|
| |
Most of the time this function is heavily cached, but when that isn't
the case fetching the counts room by room slows down push delivery on
users with many (thousands) of rooms.
Signed off by Nick @ Beeper.
|
| |
|
|
|
|
|
|
| |
This avoids doing work that will never be used (since the
resulting unread counts will never be sent in a /sync
response).
The negative of doing this is that unread counts will be
incorrect when the feature is initially enabled.
|
| |
|
|
|
|
|
| |
Broke by #13522
It looks like we have some rules in the DB with a priority class less
than 0 that don't override the base rules. Before these were just
dropped, but #13522 made that a hard error.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This improves load times for push rules:
| Version | Time per user | Time for 1k users |
| -------------------- | ------------- | ----------------- |
| Before | 138 µs | 138ms |
| Now (with custom) | 2.11 µs | 2.11ms |
| Now (without custom) | 49.7 ns | 0.05 ms |
This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
|
| |\ |
|
| | |
| |
| | |
This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078
|
| |/
|
|
| |
Signed-off-by: Sean Quah <seanq@matrix.org>
|
| | |
|
| |
|
| |
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Fixes #11887 hopefully.
The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.
This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.
In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
simplify the access token verification logic. (#12986)
This simplifies the access token verification logic by removing the `rights`
parameter which was only ever used for the unsubscribe link in email
notifications. The latter has been moved under the `/_synapse` namespace,
since it is not a standard API.
This also makes the email verification link more secure, by embedding the
app_id and pushkey in the macaroon and verifying it. This prevents the user
from tampering the query parameters of that unsubscribe link.
Macaroon generation is refactored:
- Centralised all macaroon generation and verification logic to the
`MacaroonGenerator`
- Moved to `synapse.utils`
- Changed the constructor to require only a `Clock`, hostname, and a secret key
(instead of a full `Homeserver`).
- Added tests for all methods.
|
| | |
|
| | |
|
| |
|
|
| |
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
|
| | |
|
| |
|
|
| |
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
|
| |
|
|
|
|
|
|
|
|
| |
Implements the following behind an experimental configuration flag:
* A new push rule kind for mutually related events.
* A new default push rule (`.m.rule.thread_reply`) under an unstable prefix.
This is missing part of MSC3772:
* The `.m.rule.thread_reply_to_me` push rule, this depends on MSC3664 / #11804.
|
| |
|
|
| |
accept state filters and update calls where possible (#12791)
|
| |
|
|
| |
badge_count_last_call was always zero when the response for push
notifications included a "rejected" key which mapped to an empty list.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Parse the `m.relates_to` event content field (which describes relations)
in a single place, this is used during:
* Event persistence.
* Validation of the Client-Server API.
* Fetching bundled aggregations.
* Processing of push rules.
Each of these separately implement the logic and each made slightly
different assumptions about what was valid. Some had minor / potential
bugs.
|
| |
|
|
| |
It simply passes through to `BulkPushRuleEvaluator`, which can be
called directly instead.
|
| |
|
|
|
|
|
|
|
|
| |
Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing.
The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching).
One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used.
Part of #12684
|
| |
|
|
|
|
| |
* Move `_condition_checker` into `PushRuleEvaluatorForEvent`.
* Move the condition cache into `PushRuleEvaluatorForEvent`.
* Improve docstrings.
* Inline a method which is only called once.
|
| | |
|
| |
|
|
|
|
| |
events (#12601)
Fixes vector-im/element-web#20788
Implements matrix-org/matrix-spec-proposals#3786
|
| |
|
|
|
| |
* Changes hidden read receipts to be a separate receipt type
(instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
|
| |
|
|
|
|
|
|
|
|
|
| |
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.
Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
| |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
| |
This should speed up push rule calculations for rooms with large numbers of local users when the main push rule cache fails.
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |\ |
|
| | |
| |
| |
| |
| | |
`markupsafe.Markup`" (#12296)
This reverts commit 8810c93e828a9ed1dcb008d08be8aa9fcb4d28c3.
|
| |\| |
|
| | |
| |
| |
| |
| | |
(#12289)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |/
|
|
|
| |
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
|
| |
|
| |
And rename the field used for caching from _id to _cache_key.
|
| |
|
|
|
|
|
|
|
|
| |
* Fix `PushRuleEvaluator` to work on frozendicts
frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.
* Fix event filtering for frozen events
Looks like this one was introduced by #11194.
|
| |
|
|
|
|
|
| |
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
|
| |
|
| |
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
|
| | |
|
| |
|
| |
Manually reverts the merge from cdbb8e6d6e36e0b6bc36e676d8fe66c96986b399.
|
| |
|
|
|
|
|
|
|
|
|
| |
This is some odds and ends found during the review of #11791
and while continuing to work in this code:
* Return attrs classes instead of dictionaries from some methods
to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
to not include the bundle at all, as the docstring states).
|
| |
|
|
|
| |
documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500
Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
|
| | |
|
| |
|
|
|
| |
#11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
|
| | |
|
| |
|
|
| |
This should fix pushers with an email in non-canonical form is used as
the pushkey.
|
| |
|
| |
And expand some type hints in the receipts storage module.
|
| |
|
|
|
|
| |
`matrix-python-common` (#11505) (#11527)
This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
|
| | |
|
| |
|
| |
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py
Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.
The rest of this change consists of fixups to make the newer mypy + stubs pass CI.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
| |
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
|
| | |
|
| |
|
|
|
|
|
|
| |
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.
It also improves type hints since mypy can now resolve the
types of the configuration variables.
|
| | |
|
| |
|
|
| |
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
|
| |
|
|
|
|
|
|
| |
Judging by the template, this was intended ages ago, but we never
actually passed an avatar URL to the template. So let's provide one.
Closes #1546.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
| |
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
|
| | |
|
| |
|
|
|
| |
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.
To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
|
| | |
|
| |
|
| |
Instead of mixing them with user authentication methods.
|
| |
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
| | |
|
| | |
|
| |
|
|
|
| |
hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| | |
|
| |
|
|
|
| |
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| | |
|
| |
|
|
|
|
| |
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
|
| |
|
|
|
|
| |
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
was tweaked to depend on whether there was 1 sender or multiple senders,
not based on if there was 1 room or multiple rooms.
|
| |
|
|
|
| |
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
|
| |
|
|
| |
Treat the content as untrusted and do not assume it is of
the proper form.
|
| |
|
|
|
|
| |
This allows for efficiently finding which users ignore a particular
user.
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
| |
The last stream token is always known and we do not need to handle none.
|
| |
|
|
| |
This fixes an KeyError exception, after this PR the content
is just considered unknown.
|
| |
|
| |
This improves type hinting and should use less memory.
|
| |
|
|
| |
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:
* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events
The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
|
| |
|
|
|
|
|
|
| |
per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).
However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
|
| |
|
| |
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
|
| |
|
| |
We don't always need the full power of a DeferredCache.
|
| |
|
|
|
|
|
| |
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.
This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.
I don't really know how much this will help.
|
| | |
|
| |
|
|
| |
This can happen if e.g. the room invited into is no longer on the
server (or if all users left the room).
|
| |
|
|
|
|
|
|
|
|
|
| |
* Add `DeferredCache.get_immediate` method
A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.
* Remove unused 'default' parameter to DeferredCache.get()
* another get_immediate instance
|
| |
|
|
| |
content (#8545)
|
| |
|
| |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| |
|
|
|
| |
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#8536)
* Fix outbound federaion with multiple event persisters.
We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.
Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.
* Change some interfaces to use RoomStreamToken.
By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
This converts calls like super(Foo, self) -> super().
Generated with:
sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it.
This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens.
The valid operations here are:
1. Is a position before or after a token
2. Fetching all events between two tokens
3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C.
Future PR will change the token type to a dedicated type.
|
| |
|
|
|
| |
This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset.
This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
|
| |
|
|
|
| |
`pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed.
I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
|
| |
|
|
| |
This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
|
| | |
|
| |
|
|
| |
marked unread (#8274)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fixup `ALTER TABLE` database queries
Make the new columns nullable, because doing otherwise can wedge a
server with a big database, as setting a default value rewrites the
table.
* Switch back to using the notifications count in the push badge
Clients are likely to be confused if we send a push but the badge count
is the unread messages one, and not the notifications one.
* Changelog
|
| | |
|
| | |
|
| |
|
| |
Fixes https://github.com/matrix-org/synapse/issues/6583
|
| |\
| |
| | |
With an undocumented configuration setting to enable them for specific users.
|
| | | |
|
| | | |
|
| | |\
| | |
| | |
| | | |
babolivier/new_push_rules
|
| | | | |
|
| | | | |
|
| | |/
|/| |
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
| |
This reuses the same scheme as federation sender sharding
|
| | |
|
| |
|
| |
We didn't do this for e.g. registration emails.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix spec compliance; tweaks without values are valid
(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)
* Changelog for 7766.
* Add documentation to `tweaks_for_actions`
May as well tidy up when I'm here.
* Add a test for `tweaks_for_actions`
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove obsolete comment about ancient temporary code
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Implement hack to set push priority
based on whether the tweaks indicate the event might cause
effects.
* Changelog for 7765
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Antilint
* Add tests for push priority
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Update synapse/push/httppusher.py
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Antilint
* Remove needless invites from tests.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| | |
|
| | |
|
| |
|
|
|
| |
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
|
| |\
| |
| | |
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
|
| | |\ |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | | |
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
|
| | | | |
|
| | |/
|/| |
|
| |/ |
|
| |
|
|
| |
Mainly because sometimes the email push code raises exceptions where the
stack traces have gotten lost, which is hopefully fixed by this.
|
| | |
|
| | |
|
| |
|
|
| |
variables (#6391)
|
| | |
|
| |
|
| |
add a lock to try to make this metric actually work
|
| | |
|
| |
|
|
|
| |
This would break notifications about un-named rooms when processing
notifications in a batch.
|
| |
|
|
| |
notifications. (#6966)
|
| |
|
|
| |
Ensure good comprehension hygiene using flake8-comprehensions.
|
| |
|
|
|
|
|
|
| |
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
|
| |
|
|
|
|
|
| |
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove redundant python2 support code
`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing
* Filter out pushers with corrupt data
When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.
* Improve logging when we can't start a pusher
Log the ID to help us understand the problem
* Make email pusher setup more robust
We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.
The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.
The proxy will then be used for
* push
* url previews
* phone-home stats
* recaptcha validation
* CAS auth validation
It will *not* be used for:
* Application Services
* Identity servers
* Outbound federation
* In worker configurations, connections from workers to masters
Fixes #4198.
|
| |
|
| |
* update version of black and also fix the mypy config being overridden
|
| |\
| |
| | |
Add StateGroupStorage interface
|
| | | |
|
| |/
|
| |
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
|
| | |
|
| |
|
|
|
|
|
| |
In ancient times Synapse would only send emails when it was notifying a user about a message they received...
Now it can do all sorts of neat things!
Change the logging so it's not just about notifications.
|
| | |
|
| |
|
| |
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:
* #5875
* #5876
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](https://github.com/matrix-org/synapse/commit/891afb57cbdf9867f2848341b29c75d6f35eef5a#diff-e591d42d30690ffb79f63bb726200891) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835
* #5969
* #5940
Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.
UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
|
| |
|
|
|
| |
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
|
| | |
|
| |
|
|
|
| |
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
This adds a default push rule following the proposal in
[MSC2153](https://github.com/matrix-org/matrix-doc/pull/2153).
See also https://github.com/vector-im/riot-web/issues/10208
See also https://github.com/matrix-org/matrix-js-sdk/pull/976
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Fix email notifications for unnamed rooms with multiple people
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
When we try and calculate a description for a room for with no name but
multiple other users we threw an exception (due to trying to subscript
result of `dict.values()`).
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
|
| |
|
|
|
|
|
|
|
|
| |
* Add a default .m.rule.tombstone push rule
In support of MSC1930: https://github.com/matrix-org/matrix-doc/pull/1930
* changelog
* Appease the changelog linter
|
| |\
| |
| | |
Send out emails with links to extend an account's validity period
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
| |
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.
Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
We're counting the number of push notifications, but not the number of badges;
I'd like to see if they are significant.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
|
| | |
|
| | |
|
| |
|
|
| |
... and rename it, for even more sanity
|
| |
|
|
|
| |
We don't do anything with the result, so this is needed to give this code a
logcontext.
|
| |
|
|
|
|
| |
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
|
| |
|
|
|
|
|
| |
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
|
| |
|
|
|
|
|
|
| |
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
|
| |
|
|
| |
simplifies the interface to _start_pushers
|
| |
|
|
|
| |
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
|
| |
|
|
|
| |
This is public (or at least, called from outside the class), so ought to have a
better name.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.
In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
they're not meant to be lazy (#3307)
|
| |\ |
|
| | |\
| | |
| | | |
replace some iteritems with six
|
| | | |
| | |
| | |
| | | |
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| | |/
| |
| |
| |
| |
| | |
plus a bonus b"" string I missed last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| |\
| |
| | |
make imports local
|
| | |
| |
| |
| | |
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\| |
|
| | |\
| | |
| | | |
Improve exception handling for background processes
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
|
| | |/
| |
| |
| |
| | |
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
|
| |/
|
|
|
|
| |
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Whenever an access token is invalidated, we should remove the associated
pushers.
|
| |\
| |
| | |
Remove preserve_context_over_{fn, deferred}
|
| | |
| |
| |
| |
| | |
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
|
