| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.
The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).
Commits should be independently reviewable.
|
| | |
|
| |
|
|
|
|
|
| |
https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale.
The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug.
The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
|
| | |
|
| |
|
|
| |
to them, instead of something in-memory (#9823)
|
| |
|
|
|
|
| |
- use a tuple rather than a list for the iterable that is passed into the
wrapped function, for performance
- test that we can pass an iterable and that keys are correctly deduped.
|
| |
|
|
|
| |
* Correctly ratelimit invites when creating a room
Also allow ratelimiting for more than one action at a time.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.
This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
| | |
| |
| |
| |
| | |
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
|
| | |
| |
| | |
This reverts commit e9eb3549d32a6f93d07de8dbd5e1ebe54c8d8278.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
|
| |/
|
|
|
|
| |
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.
Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
|
| |
|
|
|
| |
Only affects workers. Introduced in #9819.
Fixes #9899.
|
| |
|
| |
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
|
| |
|
|
|
|
| |
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
|
| |
|
|
|
| |
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.
Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| | |
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.
This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
|
| | |
| |
| |
| | |
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
|
| | |
| |
| |
| |
| |
| | |
handler (#9800)
This refactoring allows adding logic that uses the event context
before persisting it.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
room. (#9763)"
This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4.
The PR was prematurely merged and not yet approved.
|
| | |
| |
| |
| |
| |
| |
| | |
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
This only applies to an experimental room version, as defined in MSC3083.
|
| | |
| |
| | |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |/
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
| |
Signed-off-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
| |
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)
`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation
API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| |
|
|
|
|
|
| |
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| |\ |
|
| | |
| |
| |
| | |
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.
This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.
A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.
The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:
* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
| |
Per MSC3083.
|
| |
|
|
|
|
|
| |
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
|
| |
|
|
|
|
|
| |
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.
We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.
Fixes #9663
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.
`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
|
| | |
|
| |
|
|
|
|
|
| |
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
|
| |/ |
|
| |
|
|
|
|
|
|
|
| |
serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message.
`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.
To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
|
| |
|
|
| |
* Adds B00 to ignored checks.
* Fixes remaining issues.
|
| |
|
|
| |
Allows limiting who can login using OIDC via the claims
made from the IdP.
|
| |
|
|
|
| |
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
|
| |
|
|
|
|
|
| |
Fixes https://github.com/matrix-org/synapse/issues/9572
When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.
This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
|
| |
|
|
|
| |
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
|
| | |
|
| | |
|
| |
|
|
| |
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
|
| |
|
|
|
| |
Apple had to be special. They want a client secret which is generated from an EC key.
Fixes #9220. Also fixes #9212 while I'm here.
|
| |
|
| |
Type hint fixes due to Twisted 21.2.0 adding type hints.
|
| |
|
|
| |
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
|
| | |
|
| |
|
|
|
| |
Unfortunately this doesn't test re-joining the room since
that requires having another homeserver to query over
federation, which isn't easily doable in unit tests.
|
| |
|
| |
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
|
| | |
|
| |
|
|
| |
By consuming the response if the headers imply that the
content is too large.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519)
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment
This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`
Related to #9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
|
| |
|
| |
... otherwise, we don't get the cookie back.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
|
| |
|
|
| |
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
|
| | |
|
| |
|
|
|
|
|
| |
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.
The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.
Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
(#9402)
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.
---
When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.
It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.
This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:
* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver
We deduplicate by inserting all of those pending updates into a dictionary of the form:
```
{
server_name1: {presence_update1, ...},
server_name2: {presence_update1, presence_update2, ...}
}
```
Only after building this dict do we then start sending out presence updates.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.
Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.
This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
|
| |/
|
|
|
|
|
| |
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
|
| |
|
|
| |
(#9426)
|
| |
|
|
|
|
|
|
| |
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.
It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
|
| | |
|
| |
|
| |
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
|
| |
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
|
| | |
|
| |
|
| |
And convert some inlineDeferreds to async-friendly functions.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.27.0rc2 (2021-02-11)
==============================
Features
--------
- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
Bugfixes
--------
- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
Improved Documentation
----------------------
- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
|
| | |
| |
| |
| | |
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
|
| | |
| |
| |
| | |
This has the side-effect of being able to remove use of `inlineCallbacks`
in the test-cases for cleaner tracebacks.
|
| | | |
|
| | |
| |
| |
| | |
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
|
| |\ \
| | |
| | | |
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | | |
Signed-off-by: David Teller <davidt@element.io>
|
| |\ \ \ |
|
| | | |/
| |/| |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
|
| |\ \ \ |
|
| | |\| | |
|
| | | | | |
|
| | |/ /
|/| |
| | | |
Also add a few more IP ranges to the default blacklist.
|
| |/ / |
|
| | |
| |
| |
| |
| | |
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.
This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
was tweaked to depend on whether there was 1 sender or multiple senders,
not based on if there was 1 room or multiple rooms.
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | | |
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.
I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Fixes #8966.
* Factor out build_synapse_client_resource_tree
Start a function which will mount resources common to all workers.
* Move sso init into build_synapse_client_resource_tree
... so that we don't have to do it for each worker
* Fix SSO-login-via-a-worker
Expose the SSO login endpoints on workers, like the documentation says.
* Update workers config for new endpoints
Add documentation for endpoints recently added (#8942, #9017, #9262)
* remove submit_token from workers endpoints list
this *doesn't* work on workers (yet).
* changelog
* Add a comment about the odd path for SAML2Resource
|
| | | |
| | |
| | | |
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
|
| | | | |
|
| | |/
|/| |
|
| |\| |
|
| | |
| |
| | |
Fixes #8928.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.
A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.
Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
|
| | | |
|
| | |
| |
| |
| | |
Treat the content as untrusted and do not assume it is of
the proper form.
|
| | |
| |
| |
| |
| |
| | |
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
|
| |/ |
|
| |
|
| |
... to avoid clashes with other SSO mechanisms
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Factor out a common TestHtmlParser
Looks like I'm doing this in a few different places.
* Improve OIDC login test
Complete the OIDC login flow, rather than giving up halfway through.
* Ensure that OIDC login works with multiple OIDC providers
* Fix bugs in handling clientRedirectUrl
- don't drop duplicate query-params, or params with no value
- allow utf-8 in query-params
|
| |
|
|
|
|
|
| |
0dd2649c1 (#9112) changed the signature of `auth_via_oidc`. Meanwhile,
26d10331e (#9091) introduced a new test which relied on the old signature of
`auth_via_oidc`. The two branches were never tested together until they landed
in develop.
|
| |
|
|
|
|
| |
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is the final step for supporting multiple OIDC providers concurrently.
First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:
oidc_config:
enabled: true
issuer: "https://oidc_provider"
# etc
After:
oidc_providers:
- idp_id: prov1
issuer: "https://oidc_provider"
- idp_id: prov2
issuer: "https://another_oidc_provider"
The old format is still grandfathered in.
With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
|
| |
|
|
|
|
| |
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
|
| |
|
|
|
| |
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
|
| |
|
|
|
| |
* make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly.
* Move it to test_login.py, where I think it fits better.
|
| |
|
|
|
| |
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.
As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
|
| |\
| |
| | |
Give the user a better error when they present bad SSO creds
|
| | | |
|
| | |
| |
| |
| | |
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
|
| | | |
|
| |/
|
|
|
|
|
| |
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.
For now it's still hardcoded with a single provider.
|
| |
|
|
|
| |
* Add complete test for UI-Auth-via-SSO.
* review comments
|
| |
|
|
|
| |
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
|
| |\ |
|
| | |
| |
| |
| |
| | |
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Some light refactoring of OidcHandler, in preparation for bigger things:
* remove inheritance from deprecated BaseHandler
* add an object to hold the things that go into a session cookie
* factor out a separate class for manipulating said cookies
|
| | |
| |
| | |
This only applies if the user's data is to be erased.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the
LoggingContext and write the right entries to the request log).
Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.
In short: move get_user_agent out to a utility function.
|
| | |
| |
| |
| | |
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
|
| | | |
|
| | | |
|
| |\ \
| | |
| | | |
Add tests for the IdP picker
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | | |
black seems to want to reformat this, despite `black --check` being happy with
it :/
|
| | | | |
|
| |/ /
| |
| |
| |
| |
| | |
This allows for efficiently finding which users ignore a particular
user.
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| | |
| |
| | |
If we are lacking an optional dependency, skip the tests that rely on it.
|
| | | |
|
| | |
| |
| |
| | |
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Implement CasHandler.handle_redirect_request
... to make it match OidcHandler and SamlHandler
* Clean up interface for OidcHandler.handle_redirect_request
Make it accept `client_redirect_url=None`.
* Clean up interface for `SamlHandler.handle_redirect_request`
... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.
* Define a common protocol for SSO auth provider impls
* Give SsoIdentityProvider an ID and register them
* Combine the SSO Redirect servlets
Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.
* changelog
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739.
This PR adds a `__init__.py` and also fixes the test in a couple ways:
- Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us.
- Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint.
I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
|
| |
|
| |
This table has been unused since Synapse v1.17.0.
|
| |
|
|
| |
Tests were broken due to an API changing. The code used in Synapse
proper should be compatible with both versions already.
|
| |
|
|
|
|
|
| |
(#8756)
This adds an admin API that allows a server admin to get power in a room if a local user has power in a room. Will also invite the user if they're not in the room and its a private room. Can specify another user (rather than the admin user) to be granted power.
Co-authored-by: Matthew Hodgson <matthew@matrix.org>
|
| |
|
|
|
|
| |
This had two effects 1) it'd give the wrong answer and b) would iterate
*all* power levels in the auth chain of each event. The latter of which
can be *very* expensive for certain types of IRC bridge rooms that have
large numbers of power level changes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).
The static resources for the username picker itself (in
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res.
As the comment says, we might want to think about making them customisable, but that can be a follow-up.
Fixes #8876.
|
| | |
|
| |\ |
|
| | |
| |
| | |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | | |
|
| | |
| |
| |
| | |
If we see stale extremities while persisting events, and notice that
they don't change the result of state resolution, we drop them.
|
| | | |
|
| | |
| |
| |
| |
| | |
* Use the simple dictionary in fts for the user directory
* Clarify naming
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes a bug that deactivated users appear in the directory when their profile information was updated.
To change profile information of deactivated users is neccesary for example you will remove displayname or avatar.
But they should not appear in directory. They are deactivated.
Co-authored-by: Erik Johnston <erikj@jki.re>
|
| | | |
|
| | | |
|
| | |
| |
| | |
This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
|
| | |
| |
| | |
This improves type hinting and should use less memory.
|
| |\ \
| | |
| | | |
More preparatory refactoring of the OidcHandler tests
|
| | | |
| | |
| | |
| | |
| | |
| | | |
... so that we can test its behaviour when it raises.
Also pull it out to the top level so that I can use it from other test classes.
|
| | | |
| | |
| | |
| | | |
The tests that need this all do it already.
|
| | | |
| | |
| | |
| | |
| | | |
despite the warnings saying "don't implement get_extra_attributes", we had
implemented it, so the tests weren't doing what we thought they were.
|
| | | | |
|
| | | |
| | |
| | |
| | | |
This was never used, so let's get rid of it.
|
| | | |
| | |
| | |
| | | |
this didn't seem to be doing a lot, so remove it.
|
| | | | |
|
| | | | |
|
| |/ /
| |
| |
| |
| |
| | |
The two are equivalent, but really we want to check the HTTP result that got
returned to the channel, not the code that the Request object *intended* to
return to the channel.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* move simple_async_mock to test_utils
... so that it can be re-used
* Remove references to `SamlHandler._map_saml_response_to_user` from tests
This method is going away, so we can no longer use it as a test point. Instead,
factor out a higher-level method which takes a SAML object, and verify correct
behaviour by mocking out `AuthHandler.complete_sso_login`.
* changelog
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Remove references to handler._auth_handler
(and replace them with hs.get_auth_handler)
* Factor out a utility function for building Requests
* Remove mocks of `OidcHandler._map_userinfo_to_user`
This method is going away, so mocking it out is no longer a valid approach.
Instead, we mock out lower-level methods (eg _remote_id_from_userinfo), or
simply allow the regular implementation to proceed and update the expectations
accordingly.
* Remove references to `OidcHandler._map_userinfo_to_user` from tests
This method is going away, so we can no longer use it as a test point. Instead
we build mock "callback" requests which we pass into `handle_oidc_callback`,
and verify correct behaviour by mocking out `AuthHandler.complete_sso_login`.
|
| | |
| |
| |
| | |
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
|
| | | |
|
| | | |
|
| | |
| |
| | |
Fixes #8866
|
| | |
| |
| |
| | |
This defaults `ip_range_blacklist` to reserved IP ranges and also adds an
`ip_range_whitelist` setting to override it.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
Fixes / related to: https://github.com/matrix-org/synapse/issues/6533
This should do essentially the same thing as a robots.txt file telling robots to not index the media repo. https://developers.google.com/search/reference/robots_meta_tag
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Call set_avatar_url with target_user, not user_id
Fixes https://github.com/matrix-org/synapse/issues/8871
* Create 8872.bugfix
* Update synapse/rest/admin/users.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Testing
* Update changelog.d/8872.bugfix
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | | |
|
| | |
| |
| |
| | |
Authentication is done by checking a shared secret provided
in the Synapse configuration file.
|
| | |
| |
| |
| |
| | |
This is so that we can choose which algorithm to use based on the room ID.
|
| | |
| |
| |
| | |
Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.24.0rc2 (2020-12-04)
==============================
Bugfixes
--------
- Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))
Internal Changes
----------------
- Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
|
| | |
| |
| |
| |
| | |
(#8878)
This was broken in #8801.
|
| | |
| |
| |
| | |
The spec says we should return `M_FORBIDDEN` when someone tries to register and
registration is disabled.
|
| |\ \ |
|
| | |\ \
| | | |
| | | | |
UIA: offer only available auth flows
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
During user-interactive auth, do not offer password auth to users with no
password, nor SSO auth to users with no SSO.
Fixes #7559.
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
A few test cases were relying on being able to mount non-client servlets on the
test resource. it's better to give them their own Resources.
|
| | | |/
| | |
| | |
| | |
| | |
| | | |
Rather than using a single JsonResource, construct a resource tree, as we do in
the prod code, and allow testcases to add extra resources by overriding
`create_resource_dict`.
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | | |
This is now only used in `test_typing`, so move it there.
|
| |/ / |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:
* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events
The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
|
| |/
|
|
|
|
|
|
|
| |
(#8827)
We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as:
1. the next time a state event happens in the room the current state should correct itself;
2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and
3. we mostly use the state at an event to do important operations, which isn't affected by this.
|
| |
|
|
|
|
| |
This was broken in #8801 when abstracting code shared with OIDC.
After this change both SAML and OIDC have a concept of
grandfathering users, but with different implementations.
|
| | |
|
| |
|
|
| |
The idea here is to abstract out all the conditional code which tests which
methods a given password provider has, to provide a consistent interface.
|
| |
|
|
|
| |
(#8854)
This PR adds a `room_version` argument to the `RestHelper`'s `create_room_as` function for tests. I plan to use this for testing knocking, which currently uses an unstable room version.
|
| |
|
|
|
|
|
|
|
|
| |
The spec requires synapse to support `identifier` dicts for `m.login.password`
user-interactive auth, which it did not (instead, it required an undocumented
`user` parameter.)
To fix this properly, we need to pull the code that interprets `identifier`
into `AuthHandler.validate_login` so that it can be called from the UIA code.
Fixes #5665.
|
| |
|
| |
Fix a minor bug where we would offer "m.login.password" login if a custom auth provider supported it, even if password login was disabled.
|
| |
|
|
| |
These things seemed to be completely untested, so I added a load of tests for
them.
|
| |
|
|
| |
This test was broken by #8565. It doesn't need to set set `self.clock`
here anyway - that is done by `setUp`.
|
| |
|
|
|
|
|
|
| |
per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).
However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
|
