| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| | |
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
|
| | |
| |
| |
| | |
turns out we need to reuse this, so it's better in the config class.
|
| | |
| |
| |
| |
| | |
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
|
| | |
| |
| |
| | |
ONE DAY I WILL PURGE THE WORLD OF THIS EVIL
|
| | |
| |
| |
| |
| |
| | |
The transaction cache has some code which tries to stop it caching failures,
but if the callback function failed straight away, then things would happen
backwards and we'd end up with the failure stuck in the cache.
|
| | |
| |
| |
| |
| |
| | |
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
|
| | | |
|
| |/ |
|
| |\
| |
| | |
Make Client-Server API return 403 for invalid token
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This closes #2602
v1auth was created to account for the differences in status code between
the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid
tokens). However since those protocols were merged, this makes the r0
version/endpoint internally inconsistent, and violates the
specification for the r0 endpoint.
This might break clients that rely on this inconsistency with the
specification. This is said to affect the legacy angular reference
client. However, I feel that restoring parity with the spec is more
important. Either way, it is critical to inform developers about this
change, in case they rely on the illegal behaviour.
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\ \
| | |
| | | |
Escape label values in prometheus metrics
|
| | | | |
|
| |\ \ \ |
|
| | |/ /
| | |
| | |
| | |
| | |
| | | |
... which were making other, innocent, tests, fail.
Plus remove a spurious unittest.DEBUG which was making the output noisy.
|
| | |/
| |
| |
| | |
This was introduced in 4f2f5171
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
So, it turns out that if you have a first `Deferred` `D1`, you can add a
callback which returns another `Deferred` `D2`, and `D2` must then complete
before any further callbacks on `D1` will execute (and later callbacks on `D1`
get the *result* of `D2` rather than `D2` itself).
So, `D1` might have `called=True` (as in, it has started running its
callbacks), but any new callbacks added to `D1` won't get run until `D2`
completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield`
will 'block'.
In conclusion: some of our assumptions in `logcontext` were invalid. We need to
make sure that we don't optimise out the logcontext juggling when this
situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
|
| |\
| |
| | |
Use six.moves.urlparse
|
| | |
| |
| |
| |
| |
| | |
The imports were shuffled around a bunch in py3
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\ \ |
|
| | |\ \
| | | |
| | | | |
Make tests py3 compatible
|
| | | |/
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is a mixed commit that fixes various small issues
* print parentheses
* 01 is invalid syntax (it was octal in py2)
* [x for i in 1, 2] is invalid syntax
* six moves
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |/ /
| |
| |
| | |
This doesn't feel like a wheel we need to reinvent.
|
| |\ \
| | |
| | | |
Add some more variables to the unittest config
|
| | |/
| |
| |
| |
| |
| |
| | |
These worked accidentally before (python2 doesn't complain if you
compare incompatible types) but under py3 this blows up spectacularly
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\ \
| | |
| | | |
Use str(e) instead of e.message
|
| | |/
| |
| |
| |
| |
| |
| | |
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |/
|
|
|
|
| |
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
|
| | |
|
| |\
| |
| | |
move handling of auto_join_rooms to RegisterHandler
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work
This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.
This is related to issue #2725
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
|
| |\ \
| | |
| | | |
Add b prefixes to some strings that are bytes in py3
|
| | |/
| |
| |
| |
| |
| | |
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\ \
| | |
| | | |
Improve handling of SRV records for federation connections
|
| | |/
| |
| |
| | |
Signed-off-by: Silke Hofstra <silke@slxh.eu>
|
| |/
|
|
|
| |
Fixes an issue where a cache invalidation would invalidate *all* pending
entries, rather than just the entry that we intended to invalidate.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
It seemed to suffer from a bunch of off-by-one errors.
|
| | |
|
| |\
| |
| | |
Batch inserts into event_push_actions_staging
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |\
| |
| | |
Split event creation into a separate handler
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
|
| |\ \
| |/
|/| |
montoring metrics for number of cache evictions
|
| | | |
|
| | | |
|
| |\ \
| | |
| | | |
Ensure media is in local cache before thumbnailing
|
| | | | |
|
| | |/
|/|
| |
| |
| |
| | |
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
... instead of creating our own special SQLiteMemoryDbPool, whose purpose was a
bit of a mystery.
For some reason this makes one of the tests run slightly slower, so bump the
sleep(). Sorry.
|
| | |
| |
| |
| |
| | |
Configure the connectionpool used for unit tests to run the `on_new_connection`
function.
|
| | |
| |
| |
| | |
This isn't used, and looks thoroughly bitrotted.
|
| | | |
|
| |\ \
| | |
| | | |
Matthew's fixes to the unit tests
|
| | | |
| | |
| | |
| | | |
Extracted from https://github.com/matrix-org/synapse/pull/2820
|
| |/ /
| |
| |
| |
| |
| | |
Add federation_domain_whitelist
gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
|
| |\ \
| | |
| | | |
add registrations_require_3pid and allow_local_3pids
|
| | | | |
|
| |\ \ \
| |/ /
|/| /
| |/ |
Add decent impl of a FileConsumer
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Twisted core doesn't have a general purpose one, so we need to write one
ourselves.
Features:
- All writing happens in background thread
- Supports both push and pull producers
- Push producers get paused if the consumer falls behind
|
| |/
|
|
|
|
|
|
| |
It turns out that the only thing we use the __dict__ of LoggingContext for is
`request`, and given we create lots of LoggingContexts and then copy them every
time we do a db transaction or log line, using the __dict__ seems a bit
redundant. Let's try to optimise things by making the request attribute
explicit.
|
| |\
| |
| | |
Fix flaky test_rooms UTs
|
| | | |
|
| | |
| |
| |
| | |
pointless function is pointless
|
| | |
| |
| |
| |
| | |
Fix flakiness in the UTs caused by the user_directory being updated in the
background
|
| |/
|
|
|
| |
https://github.com/matrix-org/synapse/pull/2755 broke log-config generation,
which in turn broke the unit tests.
|
| |\
| |
| | |
Add all local users to the user_directory and optionally search them
|
| | | |
|
| | | |
|
| |/
|
|
|
| |
Instead of returning False when auth is incomplete, throw an exception which
can be caught with a wrapper.
|
| | |
|
| | |
|
| |
|
|
| |
Fix the test to pass the right number of args to the Store constructors
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
what could possibly go wrong
|
| |
|
|
|
| |
May as well do it whenever we parse a Group ID. We check the sigil and basic
structure here so it makes sense to check the grammar in the same place.
|
| |\
| |
| | |
Remove dead class
|
| | |
| |
| |
| | |
This isn't used anywhere.
|
| |\ \
| | |
| | | |
Add some tests for make_deferred_yieldable
|
| | | |
| | |
| | |
| | | |
The file under test is logcontext.py, not log_context.py
|
| | |/ |
|
| |/ |
|
| |\
| |
| | |
Fix stackoverflow and logcontexts from linearizer
|
| | |
| |
| |
| |
| |
| |
| | |
1. make it not blow out the stack when there are more than 50 things waiting
for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505.
2. Make it not mess up the log contexts.
|
| |\| |
|
| | |
| |
| |
| |
| | |
Support SRV records which point at AAAA records, as well as A records.
Fixes https://github.com/matrix-org/synapse/issues/2405
|
| | | |
|
| | |
| |
| |
| | |
Fix a bug where we could end up firing off multiple requests for server_keys
for the same server at the same time.
|
| | |
| |
| |
| |
| | |
- allows sysadmins the ability to lock down their servers so that people can't
send their users room invites.
|
| |\| |
|
| | |
| |
| |
| |
| |
| | |
I'm still unclear on what the intended behaviour for
`verify_json_objects_for_server` is, but at least I now understand the
behaviour of most of the things it calls...
|
| | | |
|
| |/ |
|
| |\
| |
| | |
Fix up user_ip replication commands
|
| | | |
|
| |/ |
|
| | |
|
| |\
| |
| | |
Fix phone home stats
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
|
| |/
|
|
|
|
|
|
|
| |
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.
By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
|
| |
|
|
|
| |
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
|
| |
|
|
|
|
|
| |
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).
Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.
|
| | |
|
| | |
|
| |\
| |
| | |
Move to using TCP replication
|
| | |
| |
| |
| |
| |
| |
| | |
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
|
| |\ \
| | |
| | | |
Speed up cached function access
|
| | | | |
|
| |\ \ \
| |/ /
|/| | |
Make AS's faster
|
| | |/ |
|
| |/
|
|
|
|
|
| |
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
|
| |\
| |
| | |
Reduce some CPU work on DB threads
|
| | |
| |
| |
| | |
This prevents unnecessary construction of lists
|
| |\ \
| | |
| | | |
push federation retry limiter down to matrixfederationclient
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | | |
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
|
| |\ \ \
| | | |
| | | | |
Fix time_bound_deferred to throw the right exception
|
| | |/ /
| | |
| | |
| | |
| | |
| | | |
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
|
| | | |
| | |
| | |
| | |
| | |
| | | |
* use a valid filter in rest/client/v2_alpha test
Signed-off-by: pik <alexander.maznev@gmail.com>
|
| | | |
| | |
| | |
| | | |
Signed-off-by: pik <alexander.maznev@gmail.com>
|
| | |/
|/|
| |
| |
| |
| | |
* add invalid filter tests
Signed-off-by: pik <alexander.maznev@gmail.com>
|
| |/
|
|
|
|
|
|
|
| |
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.
By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
|
| |\
| |
| | |
Don't send the full event json over replication
|
| | | |
|
| |/
|
|
|
|
|
|
| |
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.
Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
|
| |
|
|
| |
last_user_sync_ts
|
| | |
|
| |
|
|
|
|
| |
This was broken when device list updates were implemented, as Mailer
could no longer instantiate an AuthHandler due to a dependency on
federation sending.
|
| |
|
|
|
| |
This is because it now relies of the caches stream, which only works on
postgres. We are trying to test with sqlite.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| | |
Optimise state resolution
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.
This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
| |
Signed-off-by: Marcin Bachry <hegel666@gmail.com>
|
| |\
| |
| | |
Limit the number of events that can be created on a given room concurrently
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Make sure that a user cannot pretend to be a guest by adding 'guest = True'
caveats.
|
| | |
| |
| |
| |
| |
| |
| | |
The old test expected an incorrect wrapping due to the preview function
not using unicode properly, so it got the wrong length.
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
|
| | |
| |
| |
| | |
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.
Still TODO: removing the table from the schema.
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The 'time' caveat on the access tokens was something of a lie, since we weren't
enforcing it; more pertinently its presence stops us ever adding useful time
caveats.
Let's move in the right direction by not lying in our caveats.
|
| |/ /
| |
| |
| |
| |
| |
| | |
Since we're not doing refresh tokens any more, we should start killing off the
dead code paths. /tokenrefresh itself is a bit of a thornier subject, since
there might be apps out there using it, but we can at least not generate
refresh tokens on new logins.
|
| | | |
|
| | | |
|
| |\ \
| | |
| | |
| | | |
erikj/split_out_fed_txn
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | | |
Also make it an inclusive not exclusive filter, as the spec demands.
|
| | | | |
|
| |/ / |
|
| |/ |
|
| |\
| |
| | |
Allow clients to supply access_tokens as headers
|
| | | |
|
| |\ \
| | |
| | | |
Clarify Error codes for GET /filter/
|
| | | |
| | |
| | |
| | | |
* add tests for filter api errors
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
* add tests
Signed-off-by: Alexander Maznev <alexander.maznev@gmail.com>
|
| |\ \ \
| | | |
| | | | |
Implement pluggable password auth
|
| | |/ /
| | |
| | |
| | |
| | |
| | | |
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
|
| |\ \ \
| | | |
| | | |
| | | | |
erikj/replication_noop
|
| | | | |
| | | |
| | | |
| | | | |
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
|
| | |/ /
| | |
| | |
| | | |
synchronous
|
| | |/ |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
|
| | |
|
| | |
|
| |
|
|
|
| |
Specifically, if currently_active remains true then we should not notify
if only the last active time changes.
|
| |\
| |
| | |
Use state handler instead of get_users_in_room/get_joined_hosts
|
| | | |
|
| |\| |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This includes:
- Splitting out methods of a class into stand alone functions, to make
them easier to test.
- Adding unit tests to split out functions, testing HTML -> preview.
- Handle the fact that elements in lxml may have tail text.
|
| | |
|
| |
|
|
|
| |
login with token (as used by CAS auth) was broken by 067596d, such that it
always returned a 401.
|
| | |
|
| | |
|
| |
|
|
| |
Add an 'unsigned' section which includes the device display name.
|
| |
|
|
|
|
| |
In the situation where all of a user's devices get deleted, we want to
indicate this to a client, so we want to return an empty dictionary, rather
than nothing at all.
|
| |
|
|
|
|
|
|
|
|
|
| |
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
|
| |
|
|
|
| |
Wrap the `Requester` constructor with a function which provides sensible
defaults, and use it throughout
|
| | |
|
| |
|
|
| |
You can update the displayname of devices now.
|
| | |
|
| |\
| |
| | |
Create index on user_ips in the background
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
|
| |/
|
|
|
|
|
| |
1. Give the handler used for logging in unit tests a formatter, so that the
output is slightly more meaningful
2. Log some synapse.storage stuff, because it's useful.
|
| |
|
|
| |
Turns out I specced this to return a list of devices rather than a dict of them
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| | |
(pick up PR #938 in the hope of fixing the UTs)
|
| | |\
| | |
| | | |
Record device_id in client_ips
|
| | | |
| | |
| | |
| | |
| | | |
Record the device_id when we add a client ip; it's somewhat redundant as we
could get it via the access_token, but it will make querying rather easier.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This doesn't cover *all* of the registration flows, but it does cover the most
common ones: in particular: shared_secret registration, appservice
registration, and normal user/pass registration.
Pull device_id from the registration parameters. Register the device in the
devices table. Associate the device with the returned access and refresh
tokens. Profit.
|
| |/
|
|
|
|
|
| |
implement a GET /devices endpoint which lists all of the user's devices.
It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
|
| |\
| |
| | |
Further registration refactoring
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* `RegistrationHandler.appservice_register` no longer issues an access token:
instead it is left for the caller to do it. (There are two of these, one in
`synapse/rest/client/v1/register.py`, which now simply calls
`AuthHandler.issue_access_token`, and the other in
`synapse/rest/client/v2_alpha/register.py`, which is covered below).
* In `synapse/rest/client/v2_alpha/register.py`, move the generation of
access_tokens into `_create_registration_details`. This means that the normal
flow no longer needs to call `AuthHandler.issue_access_token`; the
shared-secret flow can tell `RegistrationHandler.register` not to generate a
token; and the appservice flow continues to work despite the above change.
|
| |\ \
| |/
|/| |
Feature: Add filter to /messages. Add 'contains_url' to filter.
|
| | | |
|
| |\ \
| | |
| | | |
rest/client/v2_alpha/register.py: Refactor flow somewhat.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is meant to be an *almost* non-functional change, with the exception that
it fixes what looks a lot like a bug in that it only calls
`auth_handler.add_threepid` and `add_pusher` once instead of three times.
The idea is to move the generation of the `access_token` out of
`registration_handler.register`, because `access_token`s now require a
device_id, and we only want to generate a device_id once registration has been
successful.
|
