| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
| |
v1.109.0. (#17428)
Introduced in: #17215
This caused us a minor bit of grief as the volume of logs produced was
much higher than normal
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
| |
|
|
|
| |
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
|
| |
|
|
|
|
|
| |
Otherwise things will get confused.
An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| | |
|
| |
|
| |
Fixes #16417
|
| | |
|
| | |
|
| |
|
|
|
| |
(#15791)
c.f. #13476
|
| | |
|
| |
|
|
|
|
|
| |
Destination was being used incorrectly (a single destination instead
of a list of destinations was being passed).
This also updates some of the types in the area to not use Collection[str],
which is a footgun.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#14870)
* Allow `AbstractSet` in `StrCollection`
Or else frozensets are excluded. This will be useful in an upcoming
commit where I plan to change a function that accepts `List[str]` to
accept `StrCollection` instead.
* `rooms_to_exclude` -> `rooms_to_exclude_globally`
I am about to make use of this exclusion mechanism to exclude rooms for
a specific user and a specific sync. This rename helps to clarify the
distinction between the global config and the rooms to exclude for a
specific sync.
* Better function names for internal sync methods
* Track a list of excluded rooms on SyncResultBuilder
I plan to feed a list of partially stated rooms for this sync to ignore
* Exclude partial state rooms during eager sync
using the mechanism established in the previous commit
* Track un-partial-state stream in sync tokens
So that we can work out which rooms have become fully-stated during a
given sync period.
* Fix mutation of `@cached` return value
This was fouling up a complement test added alongside this PR.
Excluding a room would mean the set of forgotten rooms in the cache
would be extended. This means that room could be erroneously considered
forgotten in the future.
Introduced in #12310, Synapse 1.57.0. I don't think this had any
user-visible side effects (until now).
* SyncResultBuilder: track rooms to force as newly joined
Similar plan as before. We've omitted rooms from certain sync responses;
now we establish the mechanism to reintroduce them into future syncs.
* Read new field, to present rooms as newly joined
* Force un-partial-stated rooms to be newly-joined
for eager incremental syncs only, provided they're still fully stated
* Notify user stream listeners to wake up long polling syncs
* Changelog
* Typo fix
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Unnecessary list cast
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Rephrase comment
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Another comment
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Fixup merge(?)
* Poke notifier when receiving un-partial-stated msg over replication
* Fixup merge whoops
Thanks MV :)
Co-authored-by: Mathieu Velen <mathieuv@matrix.org>
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| |
|
| |
This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
Instead of hard-coding strings in many places.
|
| | |
|
| |
|
|
|
| |
online. (#12500)
Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
|
| |
|
|
|
|
|
| |
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
|
| |
|
|
| |
(#11972)
|
| | |
|
| |
|
| |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
|
|
| |
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.
Rather, it tends to clutter up debug logs with useless information.
|
| | |
|
| | |
|
| |
|
|
|
| |
`_notify_app_services_ephemeral` (#11137)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
| |
services (#11138)
|
| | |
|
| |
|
|
|
| |
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to
follow suit.
|
| | |
|
| |
|
|
|
|
|
| |
The existing tracing reports an error each time there is a timeout, which isn't
really representative.
Additionally, we log things about the way `wait_for_events` works
(eg, the result of the callback) to the *parent* span, which is confusing.
|
| |
|
|
|
| |
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
|
| |
|
| |
This is no longer required, since we have dropped support for Python 3.5.
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
|
|
|
|
| |
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| | |
|
| | |
|
| |
|
|
|
| |
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
|
|
|
| |
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.
This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.
I don't really know how much this will help.
|
| | |
|
| |
|
| |
Optionally sends typing, presence, and read receipt information to appservices.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#8536)
* Fix outbound federaion with multiple event persisters.
We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.
Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.
* Change some interfaces to use RoomStreamToken.
By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The idea is to remove some of the places we pass around `int`, where it can represent one of two things:
1. the position of an event in the stream; or
2. a token that partitions the stream, used as part of the stream tokens.
The valid operations are then:
1. did a position happen before or after a token;
2. get all events that happened before or after a token; and
3. get all events between two tokens.
(Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
|
| |
|
|
|
| |
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it.
This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens.
The valid operations here are:
1. Is a position before or after a token
2. Fetching all events between two tokens
3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C.
Future PR will change the token type to a dedicated type.
|
| |
|
| |
This removes `SourcePaginationConfig` and `get_pagination_rows`. The reasoning behind this is that these generic classes/functions erased the types of the IDs it used (i.e. instead of passing around `StreamToken` it'd pass in e.g. `token.room_key`, which don't have uniform types).
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
For direct TCP connections we need the master to relay REMOTE_SERVER_UP
commands to the other connections so that all instances get notified
about it. The old implementation just relayed to all connections,
assuming that sending back to the original sender of the command was
safe. This is not true for redis, where commands sent get echoed back to
the sender, which was causing master to effectively infinite loop
sending and then re-receiving REMOTE_SERVER_UP commands that it sent.
The fix is to ensure that we only relay to *other* connections and not
to the connection we received the notification from.
Fixes #7334.
|
| |
|
|
|
| |
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
|
| | |
|
| | |
|
| |
|
|
|
| |
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Both (!) things that register as replication listeners do the right thing wrt
logcontexts, so this is redundant.
|
| |
|
|
| |
I guess this was used once? It's not now, anyway.
|
| | |
|
| |
|
|
|
|
|
|
| |
This ensures that its resource usage metrics get recorded somewhere rather than
getting lost.
(It also fixes an error when called from a nested logging context which
completes before the bg process)
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.
The new helper function doesn't suffer from this problem.
|
| | |
|
| |
|
|
|
|
| |
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
|
| | |
| |
| |
| | |
Twisted 16.0 doesn't have addTimeout, so let's backport it.
|
| |/
|
|
| |
This doesn't feel like a wheel we need to reinvent.
|
| |
|
|
| |
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |
|
|
|
| |
These processes take a long time compared to the request, so there is lots of
"Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
|
| |
|
|
| |
what could possibly go wrong
|
| | |
|
| | |
|
| |\
| |
| |
| | |
erikj/repl_tcp_server
|
| | |
| |
| |
| |
| | |
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
|
| | |
| |
| |
| |
| | |
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
We do this by caching the set of users a user shares rooms with.
|
| | |
|
| |
|
|
|
|
| |
There was a race condition that caused the notifier to 'miss' the
timeout notification, since there were no other checks for the timeout
this caused listeners to get stuck in a loop until something happened.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
|
| | |
|
| |
|
|
|
|
| |
Change how the notifier updates the map from room_id to user streams on
receiving a join event. Make it update the map when it notifies for the
join event, rather than using the "user_joined_room" distributor signal
|
| |
|
|
|
|
| |
Access it directly from the homeserver itself. It already wasn't
inheriting from BaseHandler storing it on the Handlers object was
already somewhat dubious.
|
| | |
|
| |
|
|
| |
so we don't accidentally mail out events people shouldn't see
|
| |
|
|
|
|
|
| |
pycharm supports them so there is no need to use the other format.
Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
If a real user attempted to first peek into one room, and then another,
their room event stream would not be woken up for events in the later
room.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Erroring causes problems when people make illegal requests, because they
don't know what limit parameter they should pass.
This is definitely buggy. It leaks message counts for rooms people don't
have permission to see, via tokens. But apparently we already
consciously decided to allow that as a team, so this preserves that
behaviour.
|
| |
|
|
| |
Needs matrix-org/sytest#125 to land first
|
| | |
|
| |
|
|
| |
Squash-merge of PR #345 from daniel/anonymousevents
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/notifier.py
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
_NotifierUserStream.new_listener that accpets a token to avoid races.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
they are notified in
|
| |
|
|
| |
the pending events by the correct room_stream_id
|
| | |
|
| |
|
|
| |
string in on_new_user_event
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
hasn't advanced for that source
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/notifier.py
|
| | |\
| | |
| | |
| | |
| | | |
Conflicts:
synapse/notifier.py
|
| | | | |
|
| | | | |
|
| |\| | |
|
| | | |
| | |
| | |
| | | |
client gets duplicate events
|
| |/ / |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
it has been called
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
when we are about to process them
|
| |
|
|
| |
they have been subscribed to. This meant that the listeners didn't clean themselves up fully from all the dicts
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
objects (users, rooms, appservices)
|
| |
|
|
| |
and appservice_*
|
| |
|
|
|
|
|
|
| |
This requires the notifier to have knowledge of appservice listeners so it can
do the regex checks on incoming invites to see if the state_key matches. It
isn't enough to just rely on the room listeners and store.get_app_service_rooms
as the room will initially not exist or won't be on the ASes radar due to
having none of its users in the room.
|
| |\ |
|
| | |
| |
| |
| |
| |
| | |
This is so that the DeferredLists actually consume the error instead of
propogating down the non-existent errback chain. This should reduce the
number of unhandled errors we are seeing.
|
| |/
|
|
| |
when poking ASes.
|
| | |
|
| | |
|
| |
|
|
| |
updates. Otherwise the timeout could fire while we are waiting for the database to return any updates it might have
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
because they don't interact well with the logging contexts
|
| | |
|
| |
|
|
| |
after it has been stashed to reduce potential for confusion
|
| | |
|
| |
|
|
| |
non-zero but sub-second waits don't collapse to zero
|
| |
|
|
| |
hasn't been incorporated in time for launch.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| | |
Conflicts:
synapse/notifier.py
webclient/room/room-controller.js
webclient/room/room.html
|
| | | |
|
| |/
|
|
| |
that source, not overall eventstream tokens
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
|
|
reflects the change in the underlying storage model.
|
