| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
|
|
| |
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.
This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
|
| | |
|
| |
|
|
| |
If we see stale extremities while persisting events, and notice that
they don't change the result of state resolution, we drop them.
|
| |
|
|
| |
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
|
| | |
|
| |
|
| |
Fixes #4042
|
| |\
| |
| | |
Fix serialisation errors when using third-party event rules.
|
| | |
| |
| |
| |
| |
| | |
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change.
For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included.
The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event.
This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem.
In essence this is just moving some inline code to a reusable store method.
|
| |/ |
|
| |
|
| |
EventBuilder.build wants auth events these days
|
| |\
| |
| | |
Support modifying event content from ThirdPartyRules modules
|
| | |
| |
| |
| | |
Support returning a new event dict from `check_event_allowed`.
|
| | |
| |
| |
| |
| | |
Rather than waiting until we handle the event, call the ThirdPartyRules check
when we fist create the event.
|
| | |
| |
| |
| |
| | |
Update `EventCreationHandler.create_event` to accept an auth_events param, and
use it in `_locally_reject_invite` instead of reinventing the wheel.
|
| |/
|
|
| |
this is always the same as requester.access_token_id.
|
| | |
|
| |
|
|
|
| |
(#8476)
Should fix #3365.
|
| |
|
|
|
| |
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.
The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.
|
| |
|
| |
All handlers now available via get_*_handler() methods on the HomeServer.
|
| | |
|
| |
|
|
| |
This is now redundant, and we can just call `handle_new_client_event` directly.
|
| | |
|
| | |
|
| |
|
|
|
| |
move the "duplicate state event" handling down into `handle_new_client_event`
where it can be shared between multiple call paths.
|
| |
|
|
|
|
|
|
| |
There's no need for it to be in the dict as well as the events table. Instead,
we store it in a separate attribute in the EventInternalMetadata object, and
populate that on load.
This means that we can rely on it being correctly populated for any event which
has been persited to the database.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The idea is to remove some of the places we pass around `int`, where it can represent one of two things:
1. the position of an event in the stream; or
2. a token that partitions the stream, used as part of the stream tokens.
The valid operations are then:
1. did a position happen before or after a token;
2. get all events that happened before or after a token; and
3. get all events between two tokens.
(Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
|
| |
|
| |
this makes it possible to use from the manhole, and seems cleaner anyway.
|
| |
|
|
|
|
| |
This is *not* ready for production yet. Caveats:
1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it.
This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens.
The valid operations here are:
1. Is a position before or after a token
2. Fetching all events between two tokens
3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C.
Future PR will change the token type to a dedicated type.
|
| |
|
|
|
| |
`pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed.
I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
|
| |
|
|
| |
This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
* Revert "Add experimental support for sharding event persister. (#8170)"
This reverts commit 82c1ee1c22a87b9e6e3179947014b0f11c0a1ac3.
* Changelog
|
| |
|
|
|
|
| |
This is *not* ready for production yet. Caveats:
1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
|
| | |
|
| |
|
|
|
|
| |
Small cleanup PR.
* Removed the unused `is_guest` argument
* Added a safeguard to a (currently) impossible code path, fixing static checking at the same time.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
If we got an error persisting an event, we would try to remove the push actions
asynchronously, which would lead to a 'Re-starting finished log context'
warning.
I don't think there's any need for this to be asynchronous.
|
| | |
|
| |
|
|
|
|
| |
I think this would have caught all the cases in
https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes
more sense here than a 403
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
Fixes #2181.
The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:
* Rather undocumented, and
* causing a *lot* of config checks
This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation.
Best to be reviewed commit-by-commit.
|
| |
|
| |
These are surprisingly expensive, and we only really need to do them at startup.
|
| | |
|
| |
|
|
|
|
|
| |
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).
Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.
People probably want to look at this commit by commit.
|
| |
|
|
|
|
|
|
|
| |
(#7497)
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.
Fixes https://github.com/matrix-org/synapse/issues/7333
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
| |
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
|
| |
|
| |
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
|
| | |
|
| |
|
|
| |
used. (#7109)
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
... and set it everywhere it's called.
while we're here, rename it for consistency with `check_user_in_room` (and to
help check that I haven't missed any instances)
|
| | |
|
| |
|
|
|
| |
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
|
| |
|
|
|
|
|
| |
These are easier to work with than the strings and we normally have one around.
This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
|
| | |
|
| | |
|
| |
|
|
| |
create_new_client_event
|
| |
|
|
| |
... to make way for a new method which just returns the event ids
|
| | |
|
| | |
|
| |
|
|
| |
Pulling things out of config is currently surprisingly expensive.
|
| |
|
|
|
|
|
|
| |
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:
* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
|
| |
|
|
|
|
|
|
| |
Purge jobs don't delete the latest event in a room in order to keep the forward extremity and not break the room. On the other hand, get_state_events, when given an at_token argument calls filter_events_for_client to know if the user can see the event that matches that (sync) token. That function uses the retention policies of the events it's given to filter out those that are too old from a client's view.
Some clients, such as Riot, when loading a room, request the list of members for the latest sync token it knows about, and get confused to the point of refusing to send any message if the server tells it that it can't get that information. This can happen very easily with the message retention feature turned on and a room with low activity so that the last event sent becomes too old according to the room's retention policy.
An easy and clean fix for that issue is to discard the room's retention policies when retrieving state.
|
| | |
|
| |
|
| |
* update version of black and also fix the mypy config being overridden
|
| |\
| |
| | |
Add StateGroupStorage interface
|
| | | |
|
| |/
|
| |
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
|
| | |
|
| |
|
| |
Fixes #5905
|
| |
|
| |
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
|
| |
This is useful to allow room admins to quickly deal with a large number
of abusive messages.
|
| |
|
| |
Co-Authored-By: Erik Johnston <erik@matrix.org>
|
| | |
|
| | |
|
| |
|
|
|
| |
We already correctly filter out such redactions, but we should also deny
them over the CS API.
|
| |
|
|
|
|
|
| |
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.
I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.
THIS IS REALLY EXPERIMENTAL.
|
| | |
|
| |\
| |
| | |
Don't bundle aggregations with events in /sync or /events or state queries
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
|
| |
Follow-up to #5124
Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
|
| | |
|
| |
|
|
| |
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
There are a number of instances where a server or admin may puppet a
user to join/leave rooms, which we don't want to fail if the user has
not consented to the privacy policy. We fix this by adding a check to
test if the requester has an associated access_token, which is used as a
proxy to answer the question of whether the action is being done on
behalf of a real request from the user.
|
| | |
|
| |
|
| |
We were logging this when it was not true.
|
| |\
| |
| |
| | |
erikj/redactions_eiah
|
| | | |
|
| | |
| |
| |
| |
| | |
This is so that everything is done in one place, making it easier to
change the event format based on room version
|
| |/ |
|
| |\
| |
| | |
Split up event validation between event and builder
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The validator was being run on the EventBuilder objects, and so the
validator only checked a subset of fields. With the upcoming
EventBuilder refactor even fewer fields will be there to validate.
To get around this we split the validation into those that can be run
against an EventBuilder and those run against a fully fledged event.
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
I found these helpful in debugging my room upgrade tests.
|
| |
|
|
|
|
|
|
| |
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
|
| |
|
|
|
|
|
| |
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
|
| | |
|
| |\
| |
| | |
Fix logcontexts for running pushers
|
| | |
| |
| |
| |
| |
| |
| | |
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| | |
This was missed during the transition from attribute to getter for
getting state from context.
|
| | | |
|
| |\|
| |
| |
| | |
erikj/client_apis_move
|
| | |
| |
| |
| |
| | |
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
|
| | |
| |
| |
| |
| | |
* give them names, to improve logging
* use a deque rather than a list for efficiency
|
| | | |
|
| | | |
|
| |/
|
|
|
| |
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\ |
|
| | |
| |
| |
| |
| | |
Make it possible to put the URI in the error message and the server notice that
get sent by the server
|
| |/
|
|
| |
... because it's shorter.
|
| |
|
|
|
|
| |
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
|
| |
|
|
| |
As we're soon going to change how topological_ordering works
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| | |
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth
As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
|
| | | |
|
| |\ \ |
|
| | |\ \
| | | |
| | | | |
reraise exceptions more carefully
|
| | | |/
| | |
| | |
| | |
| | |
| | |
| | | |
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
|
| |/
|
|
|
|
| |
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
|
| |
|
|
|
|
| |
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
|
| | |
|
| | |
|
| |
|
|
|
| |
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
Make the purge request return quickly, and allow scripts to poll for updates.
|
| | |
| |
| |
| | |
Queuing up purges doesn't sound like a good thing.
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |\ \
| | |
| | | |
Create a worker for event creation
|
| | | |
| | |
| | |
| | | |
As we want to have it run on the main synapse instance
|
| | | | |
|
| |\ \ \
| |/ /
|/| | |
delete_local_events for purge_room_history
|
| | | |
| | |
| | |
| | | |
Add a flag which makes the purger delete local events
|
| | |/
| |
| |
| | |
(beacause it deletes more than state)
|
| | |
| |
| |
| |
| |
| | |
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
|
| |
what could possibly go wrong
|
| | |
|
| | |
|
| |\
| |
| | |
Initial Group Implementation
|
| | |\ |
|
| | | | |
|
| |\ \ \
| |_|/
|/| | |
Unfreeze event before serializing with ujson
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In newer versions of https://github.com/esnme/ultrajson, ujson does not
serialize frozendicts (introduced in esnme/ultrajson@53f85b1). Although
the PyPI version is still 1.35, Fedora ships with a build from commit
esnme/ultrajson@2f1d487. This causes the serialization to fail if the
distribution-provided package is used.
This runs the event through the unfreeze utility before serializing it.
Thanks to @ignatenkobrain for tracking down the root cause.
fixes #2351
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
|
| | | |
|
| | | |
|
| | | |
|
| |/
|
| |
Demonstration of how you might add some hooks to filter out spammy events.
|
| |
|
|
|
|
|
|
| |
Since we didn't instansiate the PusherPool at start time it could fail
at run time, which it did for some users.
This may or may not fix things for those users, but it should happen at
start time and stop the server from starting.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Limit the number of events that can be created on a given room concurrently
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Move the presence handler out of the Handlers object
|
| | | |
|
| |/
|
|
| |
s/domian/domain/g
|
| | |
|
| |
|
|
|
|
|
|
| |
Wait until we sign a message to get the signing key from the homeserver
config. This means that the message handler can be created without
having a signing key in the config which means that separate processes
like the pusher that don't send messages and don't need to sign them can
still access the handlers.
|
| | |
|
| |
|
|
| |
so we don't accidentally mail out events people shouldn't see
|
| |
|
|
|
|
|
|
| |
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
|
| |
|
|
| |
collect_presencelike_data
|
| | |
|
| | |
|
| |\
| |
| | |
Add a stream for push rule updates
|
