From 22a20044280d6c0a16ad9e94baf486046d536e5c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 12 Oct 2018 17:53:14 -0600 Subject: Update documentation and templates for new consent --- docs/consent_tracking.md | 13 +++++++++---- docs/privacy_policy_templates/en/1.0.html | 15 +++++++++------ 2 files changed, 18 insertions(+), 10 deletions(-) (limited to 'docs') diff --git a/docs/consent_tracking.md b/docs/consent_tracking.md index 064eae82f7..3634d13d4f 100644 --- a/docs/consent_tracking.md +++ b/docs/consent_tracking.md @@ -31,7 +31,7 @@ Note that the templates must be stored under a name giving the language of the template - currently this must always be `en` (for "English"); internationalisation support is intended for the future. -The template for the policy itself should be versioned and named according to +The template for the policy itself should be versioned and named according to the version: for example `1.0.html`. The version of the policy which the user has agreed to is stored in the database. @@ -81,9 +81,9 @@ should be a matter of `pip install Jinja2`. On debian, try `apt-get install python-jinja2`. Once this is complete, and the server has been restarted, try visiting -`https:///_matrix/consent`. If correctly configured, this should give -an error "Missing string query parameter 'u'". It is now possible to manually -construct URIs where users can give their consent. +`https:///_matrix/consent`. If correctly configured, you should see a +default policy document. It is now possible to manually construct URIs where +users can give their consent. ### Constructing the consent URI @@ -106,6 +106,11 @@ query parameters: `https:///_matrix/consent?u=&h=68a152465a4d...`. +Note that not providing a `u` parameter will be interpreted as wanting to view +the document from an unauthenticated perspective, such as prior to registration. +Therefore, the `h` parameter is not required in this scenario. + + Sending users a server notice asking them to agree to the policy ---------------------------------------------------------------- diff --git a/docs/privacy_policy_templates/en/1.0.html b/docs/privacy_policy_templates/en/1.0.html index 55c5e4b612..321c7e4671 100644 --- a/docs/privacy_policy_templates/en/1.0.html +++ b/docs/privacy_policy_templates/en/1.0.html @@ -12,12 +12,15 @@

All your base are belong to us.

-
- - - - -
+ {% if not public_version %} + +
+ + + + +
+ {% endif %} {% endif %} -- cgit 1.4.1 From 0f5e51f726756318f355d988856730a9930e2d2f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 6 Nov 2018 03:32:34 -0700 Subject: Add config variables for enabling terms auth and the policy name (#4142) So people can still collect consent the old way if they want to. --- changelog.d/4004.feature | 2 +- changelog.d/4133.feature | 2 +- changelog.d/4142.feature | 1 + docs/consent_tracking.md | 40 ++++++++++++++++++++++++++++---- synapse/config/consent_config.py | 18 ++++++++++++++ synapse/handlers/auth.py | 2 +- synapse/rest/client/v2_alpha/register.py | 2 +- synapse/rest/consent/consent_resource.py | 2 +- tests/test_terms_auth.py | 5 ++-- tests/utils.py | 2 ++ 10 files changed, 65 insertions(+), 11 deletions(-) create mode 100644 changelog.d/4142.feature (limited to 'docs') diff --git a/changelog.d/4004.feature b/changelog.d/4004.feature index ef5cdaf5ec..89975f4c6e 100644 --- a/changelog.d/4004.feature +++ b/changelog.d/4004.feature @@ -1 +1 @@ -Add `m.login.terms` to the registration flow when consent tracking is enabled. **This makes the template arguments conditionally optional on a new `public_version` variable - update your privacy templates to support this.** +Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. diff --git a/changelog.d/4133.feature b/changelog.d/4133.feature index ef5cdaf5ec..89975f4c6e 100644 --- a/changelog.d/4133.feature +++ b/changelog.d/4133.feature @@ -1 +1 @@ -Add `m.login.terms` to the registration flow when consent tracking is enabled. **This makes the template arguments conditionally optional on a new `public_version` variable - update your privacy templates to support this.** +Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. diff --git a/changelog.d/4142.feature b/changelog.d/4142.feature new file mode 100644 index 0000000000..89975f4c6e --- /dev/null +++ b/changelog.d/4142.feature @@ -0,0 +1 @@ +Include flags to optionally add `m.login.terms` to the registration flow when consent tracking is enabled. diff --git a/docs/consent_tracking.md b/docs/consent_tracking.md index 3634d13d4f..c586b5f0b6 100644 --- a/docs/consent_tracking.md +++ b/docs/consent_tracking.md @@ -81,9 +81,40 @@ should be a matter of `pip install Jinja2`. On debian, try `apt-get install python-jinja2`. Once this is complete, and the server has been restarted, try visiting -`https:///_matrix/consent`. If correctly configured, you should see a -default policy document. It is now possible to manually construct URIs where -users can give their consent. +`https:///_matrix/consent`. If correctly configured, this should give +an error "Missing string query parameter 'u'". It is now possible to manually +construct URIs where users can give their consent. + +### Enabling consent tracking at registration + +1. Add the following to your configuration: + + ```yaml + user_consent: + require_at_registration: true + policy_name: "Privacy Policy" # or whatever you'd like to call the policy + ``` + +2. In your consent templates, make use of the `public_version` variable to + see if an unauthenticated user is viewing the page. This is typically + wrapped around the form that would be used to actually agree to the document: + + ``` + {% if not public_version %} + +
+ + + + +
+ {% endif %} + ``` + +3. Restart Synapse to apply the changes. + +Visiting `https:///_matrix/consent` should now give you a view of the privacy +document. This is what users will be able to see when registering for accounts. ### Constructing the consent URI @@ -108,7 +139,8 @@ query parameters: Note that not providing a `u` parameter will be interpreted as wanting to view the document from an unauthenticated perspective, such as prior to registration. -Therefore, the `h` parameter is not required in this scenario. +Therefore, the `h` parameter is not required in this scenario. To enable this +behaviour, set `require_at_registration` to `true` in your `user_consent` config. Sending users a server notice asking them to agree to the policy diff --git a/synapse/config/consent_config.py b/synapse/config/consent_config.py index e22c731aad..f193a090ae 100644 --- a/synapse/config/consent_config.py +++ b/synapse/config/consent_config.py @@ -42,6 +42,14 @@ DEFAULT_CONFIG = """\ # until the user consents to the privacy policy. The value of the setting is # used as the text of the error. # +# 'require_at_registration', if enabled, will add a step to the registration +# process, similar to how captcha works. Users will be required to accept the +# policy before their account is created. +# +# 'policy_name' is the display name of the policy users will see when registering +# for an account. Has no effect unless `require_at_registration` is enabled. +# Defaults to "Privacy Policy". +# # user_consent: # template_dir: res/templates/privacy # version: 1.0 @@ -54,6 +62,8 @@ DEFAULT_CONFIG = """\ # block_events_error: >- # To continue using this homeserver you must review and agree to the # terms and conditions at %(consent_uri)s +# require_at_registration: False +# policy_name: Privacy Policy # """ @@ -67,6 +77,8 @@ class ConsentConfig(Config): self.user_consent_server_notice_content = None self.user_consent_server_notice_to_guests = False self.block_events_without_consent_error = None + self.user_consent_at_registration = False + self.user_consent_policy_name = "Privacy Policy" def read_config(self, config): consent_config = config.get("user_consent") @@ -83,6 +95,12 @@ class ConsentConfig(Config): self.user_consent_server_notice_to_guests = bool(consent_config.get( "send_server_notice_to_guests", False, )) + self.user_consent_at_registration = bool(consent_config.get( + "require_at_registration", False, + )) + self.user_consent_policy_name = consent_config.get( + "policy_name", "Privacy Policy", + ) def default_config(self, **kwargs): return DEFAULT_CONFIG diff --git a/synapse/handlers/auth.py b/synapse/handlers/auth.py index 85fc1fc525..a958c45271 100644 --- a/synapse/handlers/auth.py +++ b/synapse/handlers/auth.py @@ -472,7 +472,7 @@ class AuthHandler(BaseHandler): "privacy_policy": { "version": self.hs.config.user_consent_version, "en": { - "name": "Privacy Policy", + "name": self.hs.config.user_consent_policy_name, "url": "%s/_matrix/consent?v=%s" % ( self.hs.config.public_baseurl, self.hs.config.user_consent_version, diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py index c5214330ad..0515715f7c 100644 --- a/synapse/rest/client/v2_alpha/register.py +++ b/synapse/rest/client/v2_alpha/register.py @@ -360,7 +360,7 @@ class RegisterRestServlet(RestServlet): ]) # Append m.login.terms to all flows if we're requiring consent - if self.hs.config.block_events_without_consent_error is not None: + if self.hs.config.user_consent_at_registration: new_flows = [] for flow in flows: flow.append(LoginType.TERMS) diff --git a/synapse/rest/consent/consent_resource.py b/synapse/rest/consent/consent_resource.py index c85e84b465..e0f7de5d5c 100644 --- a/synapse/rest/consent/consent_resource.py +++ b/synapse/rest/consent/consent_resource.py @@ -142,7 +142,7 @@ class ConsentResource(Resource): userhmac = None has_consented = False public_version = username == "" - if not public_version: + if not public_version or not self.hs.config.user_consent_at_registration: userhmac = parse_string(request, "h", required=True, encoding=None) self._check_hash(username, userhmac) diff --git a/tests/test_terms_auth.py b/tests/test_terms_auth.py index 7deab5266f..0b71c6feb9 100644 --- a/tests/test_terms_auth.py +++ b/tests/test_terms_auth.py @@ -42,7 +42,8 @@ class TermsTestCase(unittest.HomeserverTestCase): hs.config.enable_registration_captcha = False def test_ui_auth(self): - self.hs.config.block_events_without_consent_error = True + self.hs.config.user_consent_at_registration = True + self.hs.config.user_consent_policy_name = "My Cool Privacy Policy" self.hs.config.public_baseurl = "https://example.org" self.hs.config.user_consent_version = "1.0" @@ -66,7 +67,7 @@ class TermsTestCase(unittest.HomeserverTestCase): "policies": { "privacy_policy": { "en": { - "name": "Privacy Policy", + "name": "My Cool Privacy Policy", "url": "https://example.org/_matrix/consent?v=1.0", }, "version": "1.0" diff --git a/tests/utils.py b/tests/utils.py index 565bb60d08..67ab916f30 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -123,6 +123,8 @@ def default_config(name): config.user_directory_search_all_users = False config.user_consent_server_notice_content = None config.block_events_without_consent_error = None + config.user_consent_at_registration = False + config.user_consent_policy_name = "Privacy Policy" config.media_storage_providers = [] config.autocreate_auto_join_rooms = True config.auto_join_rooms = [] -- cgit 1.4.1 From f6cbef633284aae14fee2781b9ea735fdc537765 Mon Sep 17 00:00:00 2001 From: Aaron Raimist Date: Sun, 18 Nov 2018 12:37:56 -0600 Subject: Add a note saying you need to manually reclaim disk space People keep asking why their database hasn't gotten smaller after using this API. Signed-off-by: Aaron Raimist --- docs/admin_api/purge_history_api.rst | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'docs') diff --git a/docs/admin_api/purge_history_api.rst b/docs/admin_api/purge_history_api.rst index 2da833c827..a5c3dc8149 100644 --- a/docs/admin_api/purge_history_api.rst +++ b/docs/admin_api/purge_history_api.rst @@ -61,3 +61,11 @@ the following: } The status will be one of ``active``, ``complete``, or ``failed``. + +Reclaim disk space (Postgres) +----------------------------- + +To reclaim the disk space and return it to the operating system, you need to run +`VACUUM FULL;` on the database. + +https://www.postgresql.org/docs/current/sql-vacuum.html -- cgit 1.4.1 From de8772a655e49fc57138d91e6bb184dadeac838a Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 27 Nov 2018 03:00:33 +0100 Subject: Do a GC after each test to fix logcontext leaks (#4227) * Some words about garbage collections and logcontexts * Do a GC after each test to fix logcontext leaks This feels like an awful hack, but... * changelog --- changelog.d/4227.misc | 1 + docs/log_contexts.rst | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++- tests/unittest.py | 15 +++++++++++-- 3 files changed, 71 insertions(+), 3 deletions(-) create mode 100644 changelog.d/4227.misc (limited to 'docs') diff --git a/changelog.d/4227.misc b/changelog.d/4227.misc new file mode 100644 index 0000000000..7ebd51b6a4 --- /dev/null +++ b/changelog.d/4227.misc @@ -0,0 +1 @@ +Garbage-collect after each unit test to fix logcontext leaks \ No newline at end of file diff --git a/docs/log_contexts.rst b/docs/log_contexts.rst index 82ac4f91e5..27cde11cf7 100644 --- a/docs/log_contexts.rst +++ b/docs/log_contexts.rst @@ -163,7 +163,7 @@ the logcontext was set, this will make things work out ok: provided It's all too easy to forget to ``yield``: for instance if we forgot that ``do_some_stuff`` returned a deferred, we might plough on regardless. This leads to a mess; it will probably work itself out eventually, but not before -a load of stuff has been logged against the wrong content. (Normally, other +a load of stuff has been logged against the wrong context. (Normally, other things will break, more obviously, if you forget to ``yield``, so this tends not to be a major problem in practice.) @@ -440,3 +440,59 @@ To conclude: I think this scheme would have worked equally well, with less danger of messing it up, and probably made some more esoteric code easier to write. But again — changing the conventions of the entire Synapse codebase is not a sensible option for the marginal improvement offered. + + +A note on garbage-collection of Deferred chains +----------------------------------------------- + +It turns out that our logcontext rules do not play nicely with Deferred +chains which get orphaned and garbage-collected. + +Imagine we have some code that looks like this: + +.. code:: python + + listener_queue = [] + + def on_something_interesting(): + for d in listener_queue: + d.callback("foo") + + @defer.inlineCallbacks + def await_something_interesting(): + new_deferred = defer.Deferred() + listener_queue.append(new_deferred) + + with PreserveLoggingContext(): + yield new_deferred + +Obviously, the idea here is that we have a bunch of things which are waiting +for an event. (It's just an example of the problem here, but a relatively +common one.) + +Now let's imagine two further things happen. First of all, whatever was +waiting for the interesting thing goes away. (Perhaps the request times out, +or something *even more* interesting happens.) + +Secondly, let's suppose that we decide that the interesting thing is never +going to happen, and we reset the listener queue: + +.. code:: python + + def reset_listener_queue(): + listener_queue.clear() + +So, both ends of the deferred chain have now dropped their references, and the +deferred chain is now orphaned, and will be garbage-collected at some point. +Note that ``await_something_interesting`` is a generator function, and when +Python garbage-collects generator functions, it gives them a chance to clean +up by making the ``yield`` raise a ``GeneratorExit`` exception. In our case, +that means that the ``__exit__`` handler of ``PreserveLoggingContext`` will +carefully restore the request context, but there is now nothing waiting for +its return, so the request context is never cleared. + +To reiterate, this problem only arises when *both* ends of a deferred chain +are dropped. Dropping the the reference to a deferred you're supposed to be +calling is probably bad practice, so this doesn't actually happen too much. +Unfortunately, when it does happen, it will lead to leaked logcontexts which +are incredibly hard to track down. diff --git a/tests/unittest.py b/tests/unittest.py index a9ce57da9a..2049187fd9 100644 --- a/tests/unittest.py +++ b/tests/unittest.py @@ -13,7 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +import gc import hashlib import hmac import logging @@ -31,7 +31,7 @@ from synapse.http.server import JsonResource from synapse.http.site import SynapseRequest from synapse.server import HomeServer from synapse.types import UserID, create_requester -from synapse.util.logcontext import LoggingContextFilter +from synapse.util.logcontext import LoggingContext, LoggingContextFilter from tests.server import get_clock, make_request, render, setup_test_homeserver from tests.utils import default_config @@ -115,6 +115,17 @@ class TestCase(unittest.TestCase): logging.getLogger().setLevel(level) return orig() + @around(self) + def tearDown(orig): + ret = orig() + + # force a GC to workaround problems with deferreds leaking logcontexts when + # they are GCed (see the logcontext docs) + gc.collect() + LoggingContext.set_current_context(LoggingContext.sentinel) + + return ret + def assertObjectHasAttributes(self, attrs, obj): """Asserts that the given object has each of the attributes given, and that the value of each matches according to assertEquals.""" -- cgit 1.4.1 From d2f7c4e6b1efbdd3275d02a19220a10cf00a8f66 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Fri, 14 Dec 2018 18:20:59 +0000 Subject: create support user (#4141) Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits. --- changelog.d/4141.feature | 1 + docs/admin_api/register_api.rst | 11 ++- synapse/_scripts/register_new_matrix_user.py | 19 ++++- synapse/api/auth.py | 5 +- synapse/api/constants.py | 8 ++ synapse/handlers/register.py | 15 +++- synapse/handlers/room.py | 2 +- synapse/handlers/user_directory.py | 45 ++++++----- synapse/rest/client/v1/admin.py | 11 ++- synapse/storage/monthly_active_users.py | 30 ++++++- synapse/storage/registration.py | 38 ++++++++- .../schema/delta/53/add_user_type_to_users.sql | 19 +++++ tests/api/test_auth.py | 2 + tests/handlers/test_register.py | 30 ++++++- tests/handlers/test_user_directory.py | 91 ++++++++++++++++++++++ tests/rest/client/v1/test_admin.py | 33 ++++++-- tests/storage/test_monthly_active_users.py | 34 +++++++- tests/storage/test_registration.py | 22 ++++++ tests/unittest.py | 1 + tests/utils.py | 1 - 20 files changed, 371 insertions(+), 47 deletions(-) create mode 100644 changelog.d/4141.feature create mode 100644 synapse/storage/schema/delta/53/add_user_type_to_users.sql create mode 100644 tests/handlers/test_user_directory.py (limited to 'docs') diff --git a/changelog.d/4141.feature b/changelog.d/4141.feature new file mode 100644 index 0000000000..632d3547cb --- /dev/null +++ b/changelog.d/4141.feature @@ -0,0 +1 @@ +Special-case a support user for use in verifying behaviour of a given server. The support user does not appear in user directory or monthly active user counts. diff --git a/docs/admin_api/register_api.rst b/docs/admin_api/register_api.rst index 16d65c86b3..084e74ebf5 100644 --- a/docs/admin_api/register_api.rst +++ b/docs/admin_api/register_api.rst @@ -39,13 +39,13 @@ As an example:: } The MAC is the hex digest output of the HMAC-SHA1 algorithm, with the key being -the shared secret and the content being the nonce, user, password, and either -the string "admin" or "notadmin", each separated by NULs. For an example of -generation in Python:: +the shared secret and the content being the nonce, user, password, either the +string "admin" or "notadmin", and optionally the user_type +each separated by NULs. For an example of generation in Python:: import hmac, hashlib - def generate_mac(nonce, user, password, admin=False): + def generate_mac(nonce, user, password, admin=False, user_type=None): mac = hmac.new( key=shared_secret, @@ -59,5 +59,8 @@ generation in Python:: mac.update(password.encode('utf8')) mac.update(b"\x00") mac.update(b"admin" if admin else b"notadmin") + if user_type: + mac.update(b"\x00") + mac.update(user_type.encode('utf8')) return mac.hexdigest() diff --git a/synapse/_scripts/register_new_matrix_user.py b/synapse/_scripts/register_new_matrix_user.py index 70cecde486..4c3abf06fe 100644 --- a/synapse/_scripts/register_new_matrix_user.py +++ b/synapse/_scripts/register_new_matrix_user.py @@ -35,6 +35,7 @@ def request_registration( server_location, shared_secret, admin=False, + user_type=None, requests=_requests, _print=print, exit=sys.exit, @@ -65,6 +66,9 @@ def request_registration( mac.update(password.encode('utf8')) mac.update(b"\x00") mac.update(b"admin" if admin else b"notadmin") + if user_type: + mac.update(b"\x00") + mac.update(user_type.encode('utf8')) mac = mac.hexdigest() @@ -74,6 +78,7 @@ def request_registration( "password": password, "mac": mac, "admin": admin, + "user_type": user_type, } _print("Sending registration request...") @@ -91,7 +96,7 @@ def request_registration( _print("Success!") -def register_new_user(user, password, server_location, shared_secret, admin): +def register_new_user(user, password, server_location, shared_secret, admin, user_type): if not user: try: default_user = getpass.getuser() @@ -129,7 +134,8 @@ def register_new_user(user, password, server_location, shared_secret, admin): else: admin = False - request_registration(user, password, server_location, shared_secret, bool(admin)) + request_registration(user, password, server_location, shared_secret, + bool(admin), user_type) def main(): @@ -154,6 +160,12 @@ def main(): default=None, help="New password for user. Will prompt if omitted.", ) + parser.add_argument( + "-t", + "--user_type", + default=None, + help="User type as specified in synapse.api.constants.UserTypes", + ) admin_group = parser.add_mutually_exclusive_group() admin_group.add_argument( "-a", @@ -208,7 +220,8 @@ def main(): if args.admin or args.no_admin: admin = args.admin - register_new_user(args.user, args.password, args.server_url, secret, admin) + register_new_user(args.user, args.password, args.server_url, secret, + admin, args.user_type) if __name__ == "__main__": diff --git a/synapse/api/auth.py b/synapse/api/auth.py index 5309899703..b8a9af7158 100644 --- a/synapse/api/auth.py +++ b/synapse/api/auth.py @@ -802,9 +802,10 @@ class Auth(object): threepid should never be set at the same time. """ - # Never fail an auth check for the server notices users + # Never fail an auth check for the server notices users or support user # This can be a problem where event creation is prohibited due to blocking - if user_id == self.hs.config.server_notices_mxid: + is_support = yield self.store.is_support_user(user_id) + if user_id == self.hs.config.server_notices_mxid or is_support: return if self.hs.config.hs_disabled: diff --git a/synapse/api/constants.py b/synapse/api/constants.py index f20e0fcf0b..b7f25a42a2 100644 --- a/synapse/api/constants.py +++ b/synapse/api/constants.py @@ -119,3 +119,11 @@ KNOWN_ROOM_VERSIONS = { ServerNoticeMsgType = "m.server_notice" ServerNoticeLimitReached = "m.server_notice.usage_limit_reached" + + +class UserTypes(object): + """Allows for user type specific behaviour. With the benefit of hindsight + 'admin' and 'guest' users should also be UserTypes. Normal users are type None + """ + SUPPORT = "support" + ALL_USER_TYPES = (SUPPORT) diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py index ba39e67f6f..21c17c59a0 100644 --- a/synapse/handlers/register.py +++ b/synapse/handlers/register.py @@ -126,6 +126,7 @@ class RegistrationHandler(BaseHandler): make_guest=False, admin=False, threepid=None, + user_type=None, default_display_name=None, ): """Registers a new client on the server. @@ -141,6 +142,8 @@ class RegistrationHandler(BaseHandler): since it offers no means of associating a device_id with the access_token. Instead you should call auth_handler.issue_access_token after registration. + user_type (str|None): type of user. One of the values from + api.constants.UserTypes, or None for a normal user. default_display_name (unicode|None): if set, the new user's displayname will be set to this. Defaults to 'localpart'. Returns: @@ -190,6 +193,7 @@ class RegistrationHandler(BaseHandler): make_guest=make_guest, create_profile_with_displayname=default_display_name, admin=admin, + user_type=user_type, ) if self.hs.config.user_directory_search_all_users: @@ -242,9 +246,16 @@ class RegistrationHandler(BaseHandler): # auto-join the user to any rooms we're supposed to dump them into fake_requester = create_requester(user_id) - # try to create the room if we're the first user on the server + # try to create the room if we're the first real user on the server. Note + # that an auto-generated support user is not a real user and will never be + # the user to create the room should_auto_create_rooms = False - if self.hs.config.autocreate_auto_join_rooms: + is_support = yield self.store.is_support_user(user_id) + # There is an edge case where the first user is the support user, then + # the room is never created, though this seems unlikely and + # recoverable from given the support user being involved in the first + # place. + if self.hs.config.autocreate_auto_join_rooms and not is_support: count = yield self.store.count_all_users() should_auto_create_rooms = count == 1 for r in self.hs.config.auto_join_rooms: diff --git a/synapse/handlers/room.py b/synapse/handlers/room.py index 3928faa6e7..581e96c743 100644 --- a/synapse/handlers/room.py +++ b/synapse/handlers/room.py @@ -433,7 +433,7 @@ class RoomCreationHandler(BaseHandler): """ user_id = requester.user.to_string() - self.auth.check_auth_blocking(user_id) + yield self.auth.check_auth_blocking(user_id) if not self.spam_checker.user_may_create_room(user_id): raise SynapseError(403, "You are not permitted to create rooms") diff --git a/synapse/handlers/user_directory.py b/synapse/handlers/user_directory.py index f11b430126..3c40999338 100644 --- a/synapse/handlers/user_directory.py +++ b/synapse/handlers/user_directory.py @@ -125,9 +125,12 @@ class UserDirectoryHandler(object): """ # FIXME(#3714): We should probably do this in the same worker as all # the other changes. - yield self.store.update_profile_in_user_dir( - user_id, profile.display_name, profile.avatar_url, None, - ) + is_support = yield self.store.is_support_user(user_id) + # Support users are for diagnostics and should not appear in the user directory. + if not is_support: + yield self.store.update_profile_in_user_dir( + user_id, profile.display_name, profile.avatar_url, None, + ) @defer.inlineCallbacks def handle_user_deactivated(self, user_id): @@ -329,14 +332,7 @@ class UserDirectoryHandler(object): public_value=Membership.JOIN, ) - if change is None: - # Handle any profile changes - yield self._handle_profile_change( - state_key, room_id, prev_event_id, event_id, - ) - continue - - if not change: + if change is False: # Need to check if the server left the room entirely, if so # we might need to remove all the users in that room is_in_room = yield self.store.is_host_joined( @@ -354,16 +350,25 @@ class UserDirectoryHandler(object): else: logger.debug("Server is still in room: %r", room_id) - if change: # The user joined - event = yield self.store.get_event(event_id, allow_none=True) - profile = ProfileInfo( - avatar_url=event.content.get("avatar_url"), - display_name=event.content.get("displayname"), - ) + is_support = yield self.store.is_support_user(state_key) + if not is_support: + if change is None: + # Handle any profile changes + yield self._handle_profile_change( + state_key, room_id, prev_event_id, event_id, + ) + continue + + if change: # The user joined + event = yield self.store.get_event(event_id, allow_none=True) + profile = ProfileInfo( + avatar_url=event.content.get("avatar_url"), + display_name=event.content.get("displayname"), + ) - yield self._handle_new_user(room_id, state_key, profile) - else: # The user left - yield self._handle_remove_user(room_id, state_key) + yield self._handle_new_user(room_id, state_key, profile) + else: # The user left + yield self._handle_remove_user(room_id, state_key) else: logger.debug("Ignoring irrelevant type: %r", typ) diff --git a/synapse/rest/client/v1/admin.py b/synapse/rest/client/v1/admin.py index 41534b8c2a..82433a2aa9 100644 --- a/synapse/rest/client/v1/admin.py +++ b/synapse/rest/client/v1/admin.py @@ -23,7 +23,7 @@ from six.moves import http_client from twisted.internet import defer -from synapse.api.constants import Membership +from synapse.api.constants import Membership, UserTypes from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError from synapse.http.servlet import ( assert_params_in_dict, @@ -158,6 +158,11 @@ class UserRegisterServlet(ClientV1RestServlet): raise SynapseError(400, "Invalid password") admin = body.get("admin", None) + user_type = body.get("user_type", None) + + if user_type is not None and user_type not in UserTypes.ALL_USER_TYPES: + raise SynapseError(400, "Invalid user type") + got_mac = body["mac"] want_mac = hmac.new( @@ -171,6 +176,9 @@ class UserRegisterServlet(ClientV1RestServlet): want_mac.update(password) want_mac.update(b"\x00") want_mac.update(b"admin" if admin else b"notadmin") + if user_type: + want_mac.update(b"\x00") + want_mac.update(user_type.encode('utf8')) want_mac = want_mac.hexdigest() if not hmac.compare_digest( @@ -189,6 +197,7 @@ class UserRegisterServlet(ClientV1RestServlet): password=body["password"], admin=bool(admin), generate_token=False, + user_type=user_type, ) result = yield register._create_registration_details(user_id, body) diff --git a/synapse/storage/monthly_active_users.py b/synapse/storage/monthly_active_users.py index 479e01ddc1..d6fc8edd4c 100644 --- a/synapse/storage/monthly_active_users.py +++ b/synapse/storage/monthly_active_users.py @@ -55,9 +55,12 @@ class MonthlyActiveUsersStore(SQLBaseStore): txn, tp["medium"], tp["address"] ) + if user_id: - self.upsert_monthly_active_user_txn(txn, user_id) - reserved_user_list.append(user_id) + is_support = self.is_support_user_txn(txn, user_id) + if not is_support: + self.upsert_monthly_active_user_txn(txn, user_id) + reserved_user_list.append(user_id) else: logger.warning( "mau limit reserved threepid %s not found in db" % tp @@ -182,6 +185,18 @@ class MonthlyActiveUsersStore(SQLBaseStore): Args: user_id (str): user to add/update """ + # Support user never to be included in MAU stats. Note I can't easily call this + # from upsert_monthly_active_user_txn because then I need a _txn form of + # is_support_user which is complicated because I want to cache the result. + # Therefore I call it here and ignore the case where + # upsert_monthly_active_user_txn is called directly from + # _initialise_reserved_users reasoning that it would be very strange to + # include a support user in this context. + + is_support = yield self.is_support_user(user_id) + if is_support: + return + is_insert = yield self.runInteraction( "upsert_monthly_active_user", self.upsert_monthly_active_user_txn, user_id @@ -200,6 +215,16 @@ class MonthlyActiveUsersStore(SQLBaseStore): in a database thread rather than the main thread, and we can't call txn.call_after because txn may not be a LoggingTransaction. + We consciously do not call is_support_txn from this method because it + is not possible to cache the response. is_support_txn will be false in + almost all cases, so it seems reasonable to call it only for + upsert_monthly_active_user and to call is_support_txn manually + for cases where upsert_monthly_active_user_txn is called directly, + like _initialise_reserved_users + + In short, don't call this method with support users. (Support users + should not appear in the MAU stats). + Args: txn (cursor): user_id (str): user to add/update @@ -208,6 +233,7 @@ class MonthlyActiveUsersStore(SQLBaseStore): bool: True if a new entry was created, False if an existing one was updated. """ + # Am consciously deciding to lock the table on the basis that is ought # never be a big table and alternative approaches (batching multiple # upserts into a single txn) introduced a lot of extra complexity. diff --git a/synapse/storage/registration.py b/synapse/storage/registration.py index 3d55441e33..10c3b9757f 100644 --- a/synapse/storage/registration.py +++ b/synapse/storage/registration.py @@ -19,6 +19,7 @@ from six.moves import range from twisted.internet import defer +from synapse.api.constants import UserTypes from synapse.api.errors import Codes, StoreError from synapse.storage import background_updates from synapse.storage._base import SQLBaseStore @@ -168,7 +169,7 @@ class RegistrationStore(RegistrationWorkerStore, def register(self, user_id, token=None, password_hash=None, was_guest=False, make_guest=False, appservice_id=None, - create_profile_with_displayname=None, admin=False): + create_profile_with_displayname=None, admin=False, user_type=None): """Attempts to register an account. Args: @@ -184,6 +185,10 @@ class RegistrationStore(RegistrationWorkerStore, appservice_id (str): The ID of the appservice registering the user. create_profile_with_displayname (unicode): Optionally create a profile for the user, setting their displayname to the given value + admin (boolean): is an admin user? + user_type (str|None): type of user. One of the values from + api.constants.UserTypes, or None for a normal user. + Raises: StoreError if the user_id could not be registered. """ @@ -197,7 +202,8 @@ class RegistrationStore(RegistrationWorkerStore, make_guest, appservice_id, create_profile_with_displayname, - admin + admin, + user_type ) def _register( @@ -211,6 +217,7 @@ class RegistrationStore(RegistrationWorkerStore, appservice_id, create_profile_with_displayname, admin, + user_type, ): user_id_obj = UserID.from_string(user_id) @@ -247,6 +254,7 @@ class RegistrationStore(RegistrationWorkerStore, "is_guest": 1 if make_guest else 0, "appservice_id": appservice_id, "admin": 1 if admin else 0, + "user_type": user_type, } ) else: @@ -260,6 +268,7 @@ class RegistrationStore(RegistrationWorkerStore, "is_guest": 1 if make_guest else 0, "appservice_id": appservice_id, "admin": 1 if admin else 0, + "user_type": user_type, } ) except self.database_engine.module.IntegrityError: @@ -456,6 +465,31 @@ class RegistrationStore(RegistrationWorkerStore, defer.returnValue(res if res else False) + @cachedInlineCallbacks() + def is_support_user(self, user_id): + """Determines if the user is of type UserTypes.SUPPORT + + Args: + user_id (str): user id to test + + Returns: + Deferred[bool]: True if user is of type UserTypes.SUPPORT + """ + res = yield self.runInteraction( + "is_support_user", self.is_support_user_txn, user_id + ) + defer.returnValue(res) + + def is_support_user_txn(self, txn, user_id): + res = self._simple_select_one_onecol_txn( + txn=txn, + table="users", + keyvalues={"name": user_id}, + retcol="user_type", + allow_none=True, + ) + return True if res == UserTypes.SUPPORT else False + @defer.inlineCallbacks def user_add_threepid(self, user_id, medium, address, validated_at, added_at): yield self._simple_upsert("user_threepids", { diff --git a/synapse/storage/schema/delta/53/add_user_type_to_users.sql b/synapse/storage/schema/delta/53/add_user_type_to_users.sql new file mode 100644 index 0000000000..88ec2f83e5 --- /dev/null +++ b/synapse/storage/schema/delta/53/add_user_type_to_users.sql @@ -0,0 +1,19 @@ +/* Copyright 2018 New Vector Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* The type of the user: NULL for a regular user, or one of the constants in + * synapse.api.constants.UserTypes + */ +ALTER TABLE users ADD COLUMN user_type TEXT DEFAULT NULL; diff --git a/tests/api/test_auth.py b/tests/api/test_auth.py index 379e9c4ab1..69dc40428b 100644 --- a/tests/api/test_auth.py +++ b/tests/api/test_auth.py @@ -50,6 +50,8 @@ class AuthTestCase(unittest.TestCase): # this is overridden for the appservice tests self.store.get_app_service_by_token = Mock(return_value=None) + self.store.is_support_user = Mock(return_value=defer.succeed(False)) + @defer.inlineCallbacks def test_get_user_by_req_user_valid_token(self): user_info = {"name": self.test_user, "token_id": "ditto", "device_id": "device"} diff --git a/tests/handlers/test_register.py b/tests/handlers/test_register.py index 23bce6ee7d..eb70e1daa6 100644 --- a/tests/handlers/test_register.py +++ b/tests/handlers/test_register.py @@ -17,7 +17,8 @@ from mock import Mock from twisted.internet import defer -from synapse.api.errors import ResourceLimitError +from synapse.api.constants import UserTypes +from synapse.api.errors import ResourceLimitError, SynapseError from synapse.handlers.register import RegistrationHandler from synapse.types import RoomAlias, UserID, create_requester @@ -64,6 +65,7 @@ class RegistrationTestCase(unittest.TestCase): requester, frank.localpart, "Frankie" ) self.assertEquals(result_user_id, user_id) + self.assertTrue(result_token is not None) self.assertEquals(result_token, 'secret') @defer.inlineCallbacks @@ -82,7 +84,7 @@ class RegistrationTestCase(unittest.TestCase): requester, local_part, None ) self.assertEquals(result_user_id, user_id) - self.assertEquals(result_token, 'secret') + self.assertTrue(result_token is not None) @defer.inlineCallbacks def test_mau_limits_when_disabled(self): @@ -169,6 +171,20 @@ class RegistrationTestCase(unittest.TestCase): rooms = yield self.store.get_rooms_for_user(res[0]) self.assertEqual(len(rooms), 0) + @defer.inlineCallbacks + def test_auto_create_auto_join_rooms_when_support_user_exists(self): + room_alias_str = "#room:test" + self.hs.config.auto_join_rooms = [room_alias_str] + + self.store.is_support_user = Mock(return_value=True) + res = yield self.handler.register(localpart='support') + rooms = yield self.store.get_rooms_for_user(res[0]) + self.assertEqual(len(rooms), 0) + directory_handler = self.hs.get_handlers().directory_handler + room_alias = RoomAlias.from_string(room_alias_str) + with self.assertRaises(SynapseError): + yield directory_handler.get_association(room_alias) + @defer.inlineCallbacks def test_auto_create_auto_join_where_no_consent(self): self.hs.config.user_consent_at_registration = True @@ -179,3 +195,13 @@ class RegistrationTestCase(unittest.TestCase): yield self.handler.post_consent_actions(res[0]) rooms = yield self.store.get_rooms_for_user(res[0]) self.assertEqual(len(rooms), 0) + + @defer.inlineCallbacks + def test_register_support_user(self): + res = yield self.handler.register(localpart='user', user_type=UserTypes.SUPPORT) + self.assertTrue(self.store.is_support_user(res[0])) + + @defer.inlineCallbacks + def test_register_not_support_user(self): + res = yield self.handler.register(localpart='user') + self.assertFalse(self.store.is_support_user(res[0])) diff --git a/tests/handlers/test_user_directory.py b/tests/handlers/test_user_directory.py new file mode 100644 index 0000000000..11f2bae698 --- /dev/null +++ b/tests/handlers/test_user_directory.py @@ -0,0 +1,91 @@ +# -*- coding: utf-8 -*- +# Copyright 2018 New Vector +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from mock import Mock + +from twisted.internet import defer + +from synapse.api.constants import UserTypes +from synapse.handlers.user_directory import UserDirectoryHandler +from synapse.storage.roommember import ProfileInfo + +from tests import unittest +from tests.utils import setup_test_homeserver + + +class UserDirectoryHandlers(object): + def __init__(self, hs): + self.user_directory_handler = UserDirectoryHandler(hs) + + +class UserDirectoryTestCase(unittest.TestCase): + """ Tests the UserDirectoryHandler. """ + + @defer.inlineCallbacks + def setUp(self): + hs = yield setup_test_homeserver(self.addCleanup) + self.store = hs.get_datastore() + hs.handlers = UserDirectoryHandlers(hs) + + self.handler = hs.get_handlers().user_directory_handler + + @defer.inlineCallbacks + def test_handle_local_profile_change_with_support_user(self): + support_user_id = "@support:test" + yield self.store.register( + user_id=support_user_id, + token="123", + password_hash=None, + user_type=UserTypes.SUPPORT + ) + + yield self.handler.handle_local_profile_change(support_user_id, None) + profile = yield self.store.get_user_in_directory(support_user_id) + self.assertTrue(profile is None) + display_name = 'display_name' + + profile_info = ProfileInfo( + avatar_url='avatar_url', + display_name=display_name, + ) + regular_user_id = '@regular:test' + yield self.handler.handle_local_profile_change(regular_user_id, profile_info) + profile = yield self.store.get_user_in_directory(regular_user_id) + self.assertTrue(profile['display_name'] == display_name) + + @defer.inlineCallbacks + def test_handle_user_deactivated_support_user(self): + s_user_id = "@support:test" + self.store.register( + user_id=s_user_id, + token="123", + password_hash=None, + user_type=UserTypes.SUPPORT + ) + + self.store.remove_from_user_dir = Mock() + self.store.remove_from_user_in_public_room = Mock() + yield self.handler.handle_user_deactivated(s_user_id) + self.store.remove_from_user_dir.not_called() + self.store.remove_from_user_in_public_room.not_called() + + @defer.inlineCallbacks + def test_handle_user_deactivated_regular_user(self): + r_user_id = "@regular:test" + self.store.register(user_id=r_user_id, token="123", password_hash=None) + self.store.remove_from_user_dir = Mock() + self.store.remove_from_user_in_public_room = Mock() + yield self.handler.handle_user_deactivated(r_user_id) + self.store.remove_from_user_dir.called_once_with(r_user_id) + self.store.remove_from_user_in_public_room.assert_called_once_with(r_user_id) diff --git a/tests/rest/client/v1/test_admin.py b/tests/rest/client/v1/test_admin.py index e38eb628a9..407bf0ac4c 100644 --- a/tests/rest/client/v1/test_admin.py +++ b/tests/rest/client/v1/test_admin.py @@ -19,6 +19,7 @@ import json from mock import Mock +from synapse.api.constants import UserTypes from synapse.rest.client.v1.admin import register_servlets from tests import unittest @@ -147,7 +148,9 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): nonce = channel.json_body["nonce"] want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1) - want_mac.update(nonce.encode('ascii') + b"\x00bob\x00abc123\x00admin") + want_mac.update( + nonce.encode('ascii') + b"\x00bob\x00abc123\x00admin\x00support" + ) want_mac = want_mac.hexdigest() body = json.dumps( @@ -156,6 +159,7 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): "username": "bob", "password": "abc123", "admin": True, + "user_type": UserTypes.SUPPORT, "mac": want_mac, } ) @@ -174,7 +178,9 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): nonce = channel.json_body["nonce"] want_mac = hmac.new(key=b"shared", digestmod=hashlib.sha1) - want_mac.update(nonce.encode('ascii') + b"\x00bob\x00abc123\x00admin") + want_mac.update( + nonce.encode('ascii') + b"\x00bob\x00abc123\x00admin" + ) want_mac = want_mac.hexdigest() body = json.dumps( @@ -202,8 +208,8 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): def test_missing_parts(self): """ Synapse will complain if you don't give nonce, username, password, and - mac. Admin is optional. Additional checks are done for length and - type. + mac. Admin and user_types are optional. Additional checks are done for length + and type. """ def nonce(): @@ -260,7 +266,7 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): self.assertEqual('Invalid username', channel.json_body["error"]) # - # Username checks + # Password checks # # Must be present @@ -296,3 +302,20 @@ class UserRegisterTestCase(unittest.HomeserverTestCase): self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"]) self.assertEqual('Invalid password', channel.json_body["error"]) + + # + # user_type check + # + + # Invalid user_type + body = json.dumps({ + "nonce": nonce(), + "username": "a", + "password": "1234", + "user_type": "invalid"} + ) + request, channel = self.make_request("POST", self.url, body.encode('utf8')) + self.render(request) + + self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"]) + self.assertEqual('Invalid user type', channel.json_body["error"]) diff --git a/tests/storage/test_monthly_active_users.py b/tests/storage/test_monthly_active_users.py index 9618d57463..9605301b59 100644 --- a/tests/storage/test_monthly_active_users.py +++ b/tests/storage/test_monthly_active_users.py @@ -16,6 +16,8 @@ from mock import Mock from twisted.internet import defer +from synapse.api.constants import UserTypes + from tests.unittest import HomeserverTestCase FORTY_DAYS = 40 * 24 * 60 * 60 @@ -28,6 +30,7 @@ class MonthlyActiveUsersTestCase(HomeserverTestCase): self.store = hs.get_datastore() hs.config.limit_usage_by_mau = True hs.config.max_mau_value = 50 + # Advance the clock a bit reactor.advance(FORTY_DAYS) @@ -39,14 +42,23 @@ class MonthlyActiveUsersTestCase(HomeserverTestCase): user1_email = "user1@matrix.org" user2 = "@user2:server" user2_email = "user2@matrix.org" + user3 = "@user3:server" + user3_email = "user3@matrix.org" + threepids = [ {'medium': 'email', 'address': user1_email}, {'medium': 'email', 'address': user2_email}, + {'medium': 'email', 'address': user3_email}, ] - user_num = len(threepids) + # -1 because user3 is a support user and does not count + user_num = len(threepids) - 1 self.store.register(user_id=user1, token="123", password_hash=None) self.store.register(user_id=user2, token="456", password_hash=None) + self.store.register( + user_id=user3, token="789", + password_hash=None, user_type=UserTypes.SUPPORT + ) self.pump() now = int(self.hs.get_clock().time_msec()) @@ -60,7 +72,7 @@ class MonthlyActiveUsersTestCase(HomeserverTestCase): active_count = self.store.get_monthly_active_count() - # Test total counts + # Test total counts, ensure user3 (support user) is not counted self.assertEquals(self.get_success(active_count), user_num) # Test user is marked as active @@ -221,6 +233,24 @@ class MonthlyActiveUsersTestCase(HomeserverTestCase): count = self.store.get_registered_reserved_users_count() self.assertEquals(self.get_success(count), len(threepids)) + def test_support_user_not_add_to_mau_limits(self): + support_user_id = "@support:test" + count = self.store.get_monthly_active_count() + self.pump() + self.assertEqual(self.get_success(count), 0) + + self.store.register( + user_id=support_user_id, + token="123", + password_hash=None, + user_type=UserTypes.SUPPORT + ) + + self.store.upsert_monthly_active_user(support_user_id) + count = self.store.get_monthly_active_count() + self.pump() + self.assertEqual(self.get_success(count), 0) + def test_track_monthly_users_without_cap(self): self.hs.config.limit_usage_by_mau = False self.hs.config.mau_stats_only = True diff --git a/tests/storage/test_registration.py b/tests/storage/test_registration.py index 3dfb7b903a..cb3cc4d2e5 100644 --- a/tests/storage/test_registration.py +++ b/tests/storage/test_registration.py @@ -16,6 +16,8 @@ from twisted.internet import defer +from synapse.api.constants import UserTypes + from tests import unittest from tests.utils import setup_test_homeserver @@ -99,6 +101,26 @@ class RegistrationStoreTestCase(unittest.TestCase): user = yield self.store.get_user_by_access_token(self.tokens[0]) self.assertIsNone(user, "access token was not deleted without device_id") + @defer.inlineCallbacks + def test_is_support_user(self): + TEST_USER = "@test:test" + SUPPORT_USER = "@support:test" + + res = yield self.store.is_support_user(None) + self.assertFalse(res) + yield self.store.register(user_id=TEST_USER, token="123", password_hash=None) + res = yield self.store.is_support_user(TEST_USER) + self.assertFalse(res) + + yield self.store.register( + user_id=SUPPORT_USER, + token="456", + password_hash=None, + user_type=UserTypes.SUPPORT + ) + res = yield self.store.is_support_user(SUPPORT_USER) + self.assertTrue(res) + class TokenGenerator: def __init__(self): diff --git a/tests/unittest.py b/tests/unittest.py index 092c930396..78d2f740f9 100644 --- a/tests/unittest.py +++ b/tests/unittest.py @@ -373,6 +373,7 @@ class HomeserverTestCase(TestCase): nonce_str += b"\x00admin" else: nonce_str += b"\x00notadmin" + want_mac.update(nonce.encode('ascii') + b"\x00" + nonce_str) want_mac = want_mac.hexdigest() diff --git a/tests/utils.py b/tests/utils.py index 04796a9b30..38e689983d 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -140,7 +140,6 @@ def default_config(name): config.rc_messages_per_second = 10000 config.rc_message_burst_count = 10000 config.saml2_enabled = False - config.use_frozen_dicts = False # we need a sane default_room_version, otherwise attempts to create rooms will -- cgit 1.4.1 From 98df67a8deeb7ead6859a1900b6f25e6ebceb1e0 Mon Sep 17 00:00:00 2001 From: Krithin Sitaram Date: Sat, 29 Dec 2018 07:31:49 +0800 Subject: Remove mention of lt-cred-mech in the sample coturn config. (#4333) * Remove mention of lt-cred-mech in the sample coturn config. See https://github.com/coturn/coturn/pull/262 for more context. Also clean up some minor formatting issues while I'm here. * Add changelog. Signed-off-by: Krithin Sitaram --- changelog.d/4333.misc | 1 + docs/turn-howto.rst | 5 ++--- 2 files changed, 3 insertions(+), 3 deletions(-) create mode 100644 changelog.d/4333.misc (limited to 'docs') diff --git a/changelog.d/4333.misc b/changelog.d/4333.misc new file mode 100644 index 0000000000..43f7139a48 --- /dev/null +++ b/changelog.d/4333.misc @@ -0,0 +1 @@ +Documentation improvements for coturn setup. Contributed by Krithin Sitaram. diff --git a/docs/turn-howto.rst b/docs/turn-howto.rst index e48628ce6e..a2fc5c8820 100644 --- a/docs/turn-howto.rst +++ b/docs/turn-howto.rst @@ -40,7 +40,6 @@ You may be able to setup coturn via your package manager, or set it up manually 4. Create or edit the config file in ``/etc/turnserver.conf``. The relevant lines, with example values, are:: - lt-cred-mech use-auth-secret static-auth-secret=[your secret key here] realm=turn.myserver.org @@ -52,7 +51,7 @@ You may be able to setup coturn via your package manager, or set it up manually 5. Consider your security settings. TURN lets users request a relay which will connect to arbitrary IP addresses and ports. At the least - we recommend: + we recommend:: # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay. no-tcp-relay @@ -106,7 +105,7 @@ Your home server configuration file needs the following extra keys: to refresh credentials. The TURN REST API specification recommends one day (86400000). - 4. "turn_allow_guests": Whether to allow guest users to use the TURN + 4. "turn_allow_guests": Whether to allow guest users to use the TURN server. This is enabled by default, as otherwise VoIP will not work reliably for guests. However, it does introduce a security risk as it lets guests connect to arbitrary endpoints without having gone -- cgit 1.4.1 From 08b26afeee7b7db8c9d511cb63244927cf48ba9d Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 15:29:42 +0000 Subject: Move ACME docs to docs/ACME.rst and link from UPGRADE. --- README.rst | 69 ----------------------------------------- UPGRADE.rst | 33 +++----------------- docs/ACME.rst | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 98 deletions(-) create mode 100644 docs/ACME.rst (limited to 'docs') diff --git a/README.rst b/README.rst index 9e3d85de4c..829de0864c 100644 --- a/README.rst +++ b/README.rst @@ -225,75 +225,6 @@ If you would like to use your own certificates, you can do so by changing alternatively, you can use a reverse-proxy. Apart from port 8448 using TLS, both ports are the same in the default configuration. - -ACME setup ----------- - -Synapse v1.0 will require valid TLS certificates for communication between servers -(port ``8448`` by default) in addition to those that are client-facing (port -``443``). In the case that your `server_name` config variable is the same as -the hostname that the client connects to, then the same certificate can be -used between client and federation ports without issue. Synapse v0.99.0+ -**will provision server-to-server certificates automatically for you for -free** through `Let's Encrypt -`_ if you tell it to. - -In order for Synapse to complete the ACME challenge to provision a -certificate, it needs access to port 80. Typically listening on port 80 is -only granted to applications running as root. There are thus two solutions to -this problem. - -**Using a reverse proxy** - -A reverse proxy such as Apache or nginx allows a single process (the web -server) to listen on port 80 and proxy traffic to the appropriate program -running on your server. It is the recommended method for setting up ACME as -it allows you to use your existing webserver while also allowing Synapse to -provision certificates as needed. - -For nginx users, add the following line to your existing ``server`` block:: - - location /.well-known/acme-challenge { - proxy_pass http://localhost:8009/; - } - -For Apache, add the following to your existing webserver config:: - - ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge - -Make sure to restart/reload your webserver after making changes. - - -**Authbind** - -``authbind`` allows a program which does not run as root to bind to -low-numbered ports in a controlled way. The setup is simpler, but requires a -webserver not to already be running on port 80. **This includes every time -Synapse renews a certificate**, which may be cumbersome if you usually run a -web server on port 80. Nevertheless, if you're sure port 80 is not being used -for any other purpose then all that is necessary is the following: - -Install ``authbind``. For example, on Debian/Ubuntu:: - - sudo apt-get install authbind - -Allow ``authbind`` to bind port 80:: - - sudo touch /etc/authbind/byport/80 - sudo chmod 777 /etc/authbind/byport/80 - -When Synapse is started, use the following syntax:: - - authbind --deep - -Finally, once Synapse's is able to listen on port 80 for ACME challenge -requests, it must be told to perform ACME provisioning by setting ``enabled`` -to true under the ``acme`` section in ``homeserver.yaml``:: - - acme: - enabled: true - - Registering a user ------------------ diff --git a/UPGRADE.rst b/UPGRADE.rst index f6cdec4734..74d2452749 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -51,35 +51,10 @@ returned by the Client-Server API: Upgrading to v0.99.0 ==================== -In preparation for Synapse v1.0, you must ensure your federation TLS -certificates are verifiable by signed by a trusted root CA. - -If you do not already have a valid certificate for your domain, the easiest -way to get one is with Synapse's new ACME support, which will use the ACME -protocol to provision a certificate automatically. By default, certificates -will be obtained from the publicly trusted CA Let's Encrypt. - -For a sample configuration, please inspect the new ACME section in the example -generated config by running the ``generate-config`` executable. For example:: - - ~/synapse/env3/bin/generate-config - -You will need to provide Let's Encrypt (or another ACME provider) access to -your Synapse ACME challenge responder on port 80, at the domain of your -homeserver. This requires you to either change the port of the ACME listener -provided by Synapse to a high port and reverse proxy to it, or use a tool -like ``authbind`` to allow Synapse to listen on port 80 without root access. -(Do not run Synapse with root permissions!) - -If you are already using self-signed ceritifcates, you will need to back up -or delete them (files ``example.com.tls.crt`` and ``example.com.tls.key`` in -Synapse's root directory), Synapse's ACME implementation will not overwrite -them. - -You may wish to use alternate methods such as Certbot to obtain a certificate -from Let's Encrypt, depending on your server configuration. Of course, if you -already have a valid certificate for your homeserver's domain, that can be -placed in Synapse's config directory without the need for any ACME setup. +No special steps are required, but please be aware that you will need to +replace any self-signed certificates with those verified by a root CA before +Synapse v1.0 releases in roughly a month's time after v0.99.0. Information on +how to do so can be found at `the ACME docs `_. Upgrading to v0.34.0 ==================== diff --git a/docs/ACME.rst b/docs/ACME.rst new file mode 100644 index 0000000000..2562e85dbc --- /dev/null +++ b/docs/ACME.rst @@ -0,0 +1,98 @@ +ACME +==== + +Synapse v1.0 requires that federation TLS certificates are verifiable by a +trusted root CA. If you do not already have a valid certificate for your domain, the easiest +way to get one is with Synapse's new ACME support, which will use the ACME +protocol to provision a certificate automatically. By default, certificates +will be obtained from the publicly trusted CA Let's Encrypt. + +For a sample configuration, please inspect the new ACME section in the example +generated config by running the ``generate-config`` executable. For example:: + + ~/synapse/env3/bin/generate-config + +You will need to provide Let's Encrypt (or another ACME provider) access to +your Synapse ACME challenge responder on port 80, at the domain of your +homeserver. This requires you to either change the port of the ACME listener +provided by Synapse to a high port and reverse proxy to it, or use a tool +like ``authbind`` to allow Synapse to listen on port 80 without root access. +(Do not run Synapse with root permissions!) Detailed instructions are +available under "ACME setup" below. + +If you are already using self-signed certificates, you will need to back up +or delete them (files ``example.com.tls.crt`` and ``example.com.tls.key`` in +Synapse's root directory), Synapse's ACME implementation will not overwrite +them. + +You may wish to use alternate methods such as Certbot to obtain a certificate +from Let's Encrypt, depending on your server configuration. Of course, if you +already have a valid certificate for your homeserver's domain, that can be +placed in Synapse's config directory without the need for any ACME setup. + +ACME setup +---------- + +Synapse v1.0 will require valid TLS certificates for communication between servers +(port ``8448`` by default) in addition to those that are client-facing (port +``443``). In the case that your `server_name` config variable is the same as +the hostname that the client connects to, then the same certificate can be +used between client and federation ports without issue. Synapse v0.99.0+ +will provision server-to-server certificates automatically for you for +free through `Let's Encrypt +`_ if you tell it to. + +In order for Synapse to complete the ACME challenge to provision a +certificate, it needs access to port 80. Typically listening on port 80 is +only granted to applications running as root. There are thus two solutions to +this problem. + +**Using a reverse proxy** + +A reverse proxy such as Apache or nginx allows a single process (the web +server) to listen on port 80 and proxy traffic to the appropriate program +running on your server. It is the recommended method for setting up ACME as +it allows you to use your existing webserver while also allowing Synapse to +provision certificates as needed. + +For nginx users, add the following line to your existing ``server`` block:: + + location /.well-known/acme-challenge { + proxy_pass http://localhost:8009/; + } + +For Apache, add the following to your existing webserver config:: + + ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge + +Make sure to restart/reload your webserver after making changes. + + +**Authbind** + +``authbind`` allows a program which does not run as root to bind to +low-numbered ports in a controlled way. The setup is simpler, but requires a +webserver not to already be running on port 80. **This includes every time +Synapse renews a certificate**, which may be cumbersome if you usually run a +web server on port 80. Nevertheless, if you're sure port 80 is not being used +for any other purpose then all that is necessary is the following: + +Install ``authbind``. For example, on Debian/Ubuntu:: + + sudo apt-get install authbind + +Allow ``authbind`` to bind port 80:: + + sudo touch /etc/authbind/byport/80 + sudo chmod 777 /etc/authbind/byport/80 + +When Synapse is started, use the following syntax:: + + authbind --deep + +Finally, once Synapse's is able to listen on port 80 for ACME challenge +requests, it must be told to perform ACME provisioning by setting ``enabled`` +to true under the ``acme`` section in ``homeserver.yaml``:: + + acme: + enabled: true -- cgit 1.4.1 From cbdc01cc3b826879b43ad4e8e9c1acacc60cd34b Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 15:38:27 +0000 Subject: Convert ACME docs to md --- README.rst | 2 +- UPGRADE.rst | 2 +- docs/ACME.rst | 98 ----------------------------------------------------------- 3 files changed, 2 insertions(+), 100 deletions(-) delete mode 100644 docs/ACME.rst (limited to 'docs') diff --git a/README.rst b/README.rst index 47f8ef62db..b7b3b20159 100644 --- a/README.rst +++ b/README.rst @@ -229,7 +229,7 @@ ACME setup ---------- For details on having Synapse manage your federation TLS certificates -automatically, please see ``_. +automatically, please see ``_. Registering a user ------------------ diff --git a/UPGRADE.rst b/UPGRADE.rst index 74d2452749..948867f189 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -54,7 +54,7 @@ Upgrading to v0.99.0 No special steps are required, but please be aware that you will need to replace any self-signed certificates with those verified by a root CA before Synapse v1.0 releases in roughly a month's time after v0.99.0. Information on -how to do so can be found at `the ACME docs `_. +how to do so can be found at `the ACME docs `_. Upgrading to v0.34.0 ==================== diff --git a/docs/ACME.rst b/docs/ACME.rst deleted file mode 100644 index 2562e85dbc..0000000000 --- a/docs/ACME.rst +++ /dev/null @@ -1,98 +0,0 @@ -ACME -==== - -Synapse v1.0 requires that federation TLS certificates are verifiable by a -trusted root CA. If you do not already have a valid certificate for your domain, the easiest -way to get one is with Synapse's new ACME support, which will use the ACME -protocol to provision a certificate automatically. By default, certificates -will be obtained from the publicly trusted CA Let's Encrypt. - -For a sample configuration, please inspect the new ACME section in the example -generated config by running the ``generate-config`` executable. For example:: - - ~/synapse/env3/bin/generate-config - -You will need to provide Let's Encrypt (or another ACME provider) access to -your Synapse ACME challenge responder on port 80, at the domain of your -homeserver. This requires you to either change the port of the ACME listener -provided by Synapse to a high port and reverse proxy to it, or use a tool -like ``authbind`` to allow Synapse to listen on port 80 without root access. -(Do not run Synapse with root permissions!) Detailed instructions are -available under "ACME setup" below. - -If you are already using self-signed certificates, you will need to back up -or delete them (files ``example.com.tls.crt`` and ``example.com.tls.key`` in -Synapse's root directory), Synapse's ACME implementation will not overwrite -them. - -You may wish to use alternate methods such as Certbot to obtain a certificate -from Let's Encrypt, depending on your server configuration. Of course, if you -already have a valid certificate for your homeserver's domain, that can be -placed in Synapse's config directory without the need for any ACME setup. - -ACME setup ----------- - -Synapse v1.0 will require valid TLS certificates for communication between servers -(port ``8448`` by default) in addition to those that are client-facing (port -``443``). In the case that your `server_name` config variable is the same as -the hostname that the client connects to, then the same certificate can be -used between client and federation ports without issue. Synapse v0.99.0+ -will provision server-to-server certificates automatically for you for -free through `Let's Encrypt -`_ if you tell it to. - -In order for Synapse to complete the ACME challenge to provision a -certificate, it needs access to port 80. Typically listening on port 80 is -only granted to applications running as root. There are thus two solutions to -this problem. - -**Using a reverse proxy** - -A reverse proxy such as Apache or nginx allows a single process (the web -server) to listen on port 80 and proxy traffic to the appropriate program -running on your server. It is the recommended method for setting up ACME as -it allows you to use your existing webserver while also allowing Synapse to -provision certificates as needed. - -For nginx users, add the following line to your existing ``server`` block:: - - location /.well-known/acme-challenge { - proxy_pass http://localhost:8009/; - } - -For Apache, add the following to your existing webserver config:: - - ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge - -Make sure to restart/reload your webserver after making changes. - - -**Authbind** - -``authbind`` allows a program which does not run as root to bind to -low-numbered ports in a controlled way. The setup is simpler, but requires a -webserver not to already be running on port 80. **This includes every time -Synapse renews a certificate**, which may be cumbersome if you usually run a -web server on port 80. Nevertheless, if you're sure port 80 is not being used -for any other purpose then all that is necessary is the following: - -Install ``authbind``. For example, on Debian/Ubuntu:: - - sudo apt-get install authbind - -Allow ``authbind`` to bind port 80:: - - sudo touch /etc/authbind/byport/80 - sudo chmod 777 /etc/authbind/byport/80 - -When Synapse is started, use the following syntax:: - - authbind --deep - -Finally, once Synapse's is able to listen on port 80 for ACME challenge -requests, it must be told to perform ACME provisioning by setting ``enabled`` -to true under the ``acme`` section in ``homeserver.yaml``:: - - acme: - enabled: true -- cgit 1.4.1 From ffcbd80982ad4164eda38c45d8b367b1748904c4 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 15:50:18 +0000 Subject: Actually add ACME docs --- docs/ACME.md | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/ACME.md (limited to 'docs') diff --git a/docs/ACME.md b/docs/ACME.md new file mode 100644 index 0000000000..f1a0c25697 --- /dev/null +++ b/docs/ACME.md @@ -0,0 +1,107 @@ +# ACME + +Synapse v1.0 requires that federation TLS certificates are verifiable by a +trusted root CA. If you do not already have a valid certificate for your domain, the easiest +way to get one is with Synapse's new ACME support, which will use the ACME +protocol to provision a certificate automatically. By default, certificates +will be obtained from the publicly trusted CA Let's Encrypt. + +For a sample configuration, please inspect the new ACME section in the example +generated config by running the `generate-config` executable. For example:: + + ~/synapse/env3/bin/generate-config + +You will need to provide Let's Encrypt (or another ACME provider) access to +your Synapse ACME challenge responder on port 80, at the domain of your +homeserver. This requires you to either change the port of the ACME listener +provided by Synapse to a high port and reverse proxy to it, or use a tool +like `authbind` to allow Synapse to listen on port 80 without root access. +(Do not run Synapse with root permissions!) Detailed instructions are +available under "ACME setup" below. + +If you are already using self-signed certificates, you will need to back up +or delete them (files `example.com.tls.crt` and `example.com.tls.key` in +Synapse's root directory), Synapse's ACME implementation will not overwrite +them. + +You may wish to use alternate methods such as Certbot to obtain a certificate +from Let's Encrypt, depending on your server configuration. Of course, if you +already have a valid certificate for your homeserver's domain, that can be +placed in Synapse's config directory without the need for any ACME setup. + +## ACME setup + +Synapse v1.0 will require valid TLS certificates for communication between servers +(port `8448` by default) in addition to those that are client-facing (port +`443`). In the case that your `server_name` config variable is the same as +the hostname that the client connects to, then the same certificate can be +used between client and federation ports without issue. Synapse v0.99.0+ +**will provision server-to-server certificates automatically for you for +free** through [Let's Encrypt](https://letsencrypt.org/) if you tell it to. + +In order for Synapse to complete the ACME challenge to provision a +certificate, it needs access to port 80. Typically listening on port 80 is +only granted to applications running as root. There are thus two solutions to +this problem. + +### Using a reverse proxy + +A reverse proxy such as Apache or nginx allows a single process (the web +server) to listen on port 80 and proxy traffic to the appropriate program +running on your server. It is the recommended method for setting up ACME as +it allows you to use your existing webserver while also allowing Synapse to +provision certificates as needed. + +For nginx users, add the following line to your existing `server` block: + +``` +location /.well-known/acme-challenge { + proxy_pass http://localhost:8009/; +} +``` + +For Apache, add the following to your existing webserver config:: + +``` +ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge +``` + +Make sure to restart/reload your webserver after making changes. + + +### Authbind + +`authbind` allows a program which does not run as root to bind to +low-numbered ports in a controlled way. The setup is simpler, but requires a +webserver not to already be running on port 80. **This includes every time +Synapse renews a certificate**, which may be cumbersome if you usually run a +web server on port 80. Nevertheless, if you're sure port 80 is not being used +for any other purpose then all that is necessary is the following: + +Install `authbind`. For example, on Debian/Ubuntu: + +``` +sudo apt-get install authbind +``` + +Allow `authbind` to bind port 80: + +``` +sudo touch /etc/authbind/byport/80 +sudo chmod 777 /etc/authbind/byport/80 +``` + +When Synapse is started, use the following syntax:: + +``` +authbind --deep +``` + +Finally, once Synapse is able to listen on port 80 for ACME challenge +requests, it must be told to perform ACME provisioning by setting `enabled` +to true under the `acme` section in `homeserver.yaml`: + +``` +acme: + enabled: true +``` \ No newline at end of file -- cgit 1.4.1 From 13828f7d5811c6c0de1ccc4d734a68d01a004dec Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 5 Feb 2019 16:46:28 +0000 Subject: Update docs/ACME.md Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com> --- docs/ACME.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/docs/ACME.md b/docs/ACME.md index f1a0c25697..309296cc0b 100644 --- a/docs/ACME.md +++ b/docs/ACME.md @@ -2,7 +2,7 @@ Synapse v1.0 requires that federation TLS certificates are verifiable by a trusted root CA. If you do not already have a valid certificate for your domain, the easiest -way to get one is with Synapse's new ACME support, which will use the ACME +way to get one is with Synapse's ACME support (new as of Synapse 0.99), which will use the ACME protocol to provision a certificate automatically. By default, certificates will be obtained from the publicly trusted CA Let's Encrypt. @@ -104,4 +104,4 @@ to true under the `acme` section in `homeserver.yaml`: ``` acme: enabled: true -``` \ No newline at end of file +``` -- cgit 1.4.1 From 2ca63df83b49599613b3801be2577a1d869a918b Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 16:50:00 +0000 Subject: Update ACME --- docs/ACME.md | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) (limited to 'docs') diff --git a/docs/ACME.md b/docs/ACME.md index f1a0c25697..341044dac1 100644 --- a/docs/ACME.md +++ b/docs/ACME.md @@ -1,15 +1,23 @@ # ACME -Synapse v1.0 requires that federation TLS certificates are verifiable by a -trusted root CA. If you do not already have a valid certificate for your domain, the easiest -way to get one is with Synapse's new ACME support, which will use the ACME -protocol to provision a certificate automatically. By default, certificates -will be obtained from the publicly trusted CA Let's Encrypt. +Synapse v1.0 will require valid TLS certificates for communication between +servers (port `8448` by default) in addition to those that are client-facing +(port `443`). If you do not already have a valid certificate for your domain, +the easiest way to get one is with Synapse's new ACME support, which will use +the ACME protocol to provision a certificate automatically. Synapse v0.99.0+ +will provision server-to-server certificates automatically for you for free +through [Let's Encrypt](https://letsencrypt.org/) if you tell it to. + +In the case that your `server_name` config variable is the same as +the hostname that the client connects to, then the same certificate can be +used between client and federation ports without issue. For a sample configuration, please inspect the new ACME section in the example -generated config by running the `generate-config` executable. For example:: +generated config by running the `generate-config` executable. For example: - ~/synapse/env3/bin/generate-config +``` +~/synapse/env3/bin/generate-config +``` You will need to provide Let's Encrypt (or another ACME provider) access to your Synapse ACME challenge responder on port 80, at the domain of your @@ -31,13 +39,6 @@ placed in Synapse's config directory without the need for any ACME setup. ## ACME setup -Synapse v1.0 will require valid TLS certificates for communication between servers -(port `8448` by default) in addition to those that are client-facing (port -`443`). In the case that your `server_name` config variable is the same as -the hostname that the client connects to, then the same certificate can be -used between client and federation ports without issue. Synapse v0.99.0+ -**will provision server-to-server certificates automatically for you for -free** through [Let's Encrypt](https://letsencrypt.org/) if you tell it to. In order for Synapse to complete the ACME challenge to provision a certificate, it needs access to port 80. Typically listening on port 80 is @@ -97,6 +98,8 @@ When Synapse is started, use the following syntax:: authbind --deep ``` +## Config file editing + Finally, once Synapse is able to listen on port 80 for ACME challenge requests, it must be told to perform ACME provisioning by setting `enabled` to true under the `acme` section in `homeserver.yaml`: -- cgit 1.4.1 From a6345009f92d8ffef4bd0d42196d18eac9b9bf38 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 5 Feb 2019 17:04:34 +0000 Subject: Add TL;DR and final step details to ACME --- docs/ACME.md | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) (limited to 'docs') diff --git a/docs/ACME.md b/docs/ACME.md index 15752ad9c9..8fb2bd66a9 100644 --- a/docs/ACME.md +++ b/docs/ACME.md @@ -39,13 +39,23 @@ placed in Synapse's config directory without the need for any ACME setup. ## ACME setup +The main steps for enabling ACME support in short summary are: + +1. Allow Synapse to listen on port 80 with authbind, or forward it from a reverse-proxy. +1. Set `acme:enabled` to `true` in homeserver.yaml. +1. Move your old certificates (files `example.com.tls.crt` and `example.com.tls.key` out of the way if they currently exist at the paths specified in `homeserver.yaml`. +1. Restart Synapse + +Detailed instructions for each step are provided below. + +### Listening on port 80 In order for Synapse to complete the ACME challenge to provision a certificate, it needs access to port 80. Typically listening on port 80 is only granted to applications running as root. There are thus two solutions to this problem. -### Using a reverse proxy +#### Using a reverse proxy A reverse proxy such as Apache or nginx allows a single process (the web server) to listen on port 80 and proxy traffic to the appropriate program @@ -70,7 +80,7 @@ ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-cha Make sure to restart/reload your webserver after making changes. -### Authbind +#### Authbind `authbind` allows a program which does not run as root to bind to low-numbered ports in a controlled way. The setup is simpler, but requires a @@ -98,9 +108,9 @@ When Synapse is started, use the following syntax:: authbind --deep ``` -## Config file editing +### Config file editing -Finally, once Synapse is able to listen on port 80 for ACME challenge +Once Synapse is able to listen on port 80 for ACME challenge requests, it must be told to perform ACME provisioning by setting `enabled` to true under the `acme` section in `homeserver.yaml`: @@ -108,3 +118,9 @@ to true under the `acme` section in `homeserver.yaml`: acme: enabled: true ``` + +### Starting synapse + +Ensure that the certificate paths specified in `homeserver.yaml` (`tls_certificate_path` and `tls_private_key_path`) do not currently point to any files. Synapse will not provision certificates if files exist, as it does not want to overwrite existing certificates. + +Finally, start/restart Synapse. \ No newline at end of file -- cgit 1.4.1 From 6585ef47991478ed1617f78f992b35b63475a2d7 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Tue, 5 Feb 2019 17:19:28 +0000 Subject: Neilj/1711faq (#4572) MSC1711 certificates FAQ --- UPGRADE.rst | 2 + changelog.d/4572.misc | 1 + docs/MSC1711_certificates_FAQ.md | 260 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 263 insertions(+) create mode 100644 changelog.d/4572.misc create mode 100644 docs/MSC1711_certificates_FAQ.md (limited to 'docs') diff --git a/UPGRADE.rst b/UPGRADE.rst index c46f70f699..7bd631f14c 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -80,6 +80,8 @@ from Let's Encrypt, depending on your server configuration. Of course, if you already have a valid certificate for your homeserver's domain, that can be placed in Synapse's config directory without the need for ACME. +For more information on configuring TLS certificates see the `FAQ `_ + Upgrading to v0.34.0 ==================== diff --git a/changelog.d/4572.misc b/changelog.d/4572.misc new file mode 100644 index 0000000000..ea5d49b706 --- /dev/null +++ b/changelog.d/4572.misc @@ -0,0 +1 @@ +FAQ to help server admins configure TLS certs in 0.99.0 diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md new file mode 100644 index 0000000000..0dcef57733 --- /dev/null +++ b/docs/MSC1711_certificates_FAQ.md @@ -0,0 +1,260 @@ +# MSC 1711 Certificates FAQ + +The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It +supports the r0.1 release of the server to server specification, but is +compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well +as post-r0.1 behaviour, in order to allow for a smooth upgrade across the +federation. + +The most important thing to know is that Synapse 1.0.0 will require a valid TLS +certificate on federation endpoints. Self signed certificates will not be +sufficient. + +Synapse 0.99.0 makes it easy to configure TLS certificates and will +interoperate with both >= 1.0.0 servers as well as existing servers yet to +upgrade. + +**It is critical that all admins upgrade to 0.99.0 and configure a valid TLS +certificate.** Admins will have 1 month to do so, after which 1.0.0 will be +released and those servers without a valid certificate will not longer be able +to federate with >= 1.0.0 servers. + +If you are unable to generate a valid TLS certificate for your server (e.g. +because you run it on behalf of someone who doesn't want to give you a TLS +certificate for their domain, or simply because the matrix domain is hosted on +a different server), then you can now create a /.well-known/matrix/server file +on the matrix domain in order to delegate Matrix hosting to another domain. + Admins who currently use SRV records to delegate a domain **which they do not +control TLS for** will need to switch to using .well-known/matrix/server - though +they should retain their SRV record while the federation upgrades over the +course of the month.  Other SRV records are unaffected. + +Full upgrade notes can be found in +[UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst). +What follows is a timeline and some frequently asked questions. + +For more details and context on the release of the r0.1 Server/Server API and +imminent Matrix 1.0 release, you can also see our +[main talk from FOSDEM 2019](https://matrix.org/blog/2019/02/04/matrix-at-fosdem-2019/). + +## Contents +* Timeline +* Synapse 0.99.0 has just been released, what do I need to do right now? +* How do I upgrade? +* What will happen if I do not set up a valid federation certificate + immediately? +* What will happen if I do nothing at all? +* When do I need a SRV record or .well-known URI? +* Can I still use an SRV record? +* I have created a .well-known URI. Do I still need an SRV record? +* It used to work just fine, why are you breaking everything? +* Can I manage my own certificates rather than having Synapse renew + certificates itself? +* Do you still recommend against using a reverse-proxy on the federation port? +* Do I still need to give my TLS certificates to Synapse if I am using a + reverse-proxy? +* Do I need the same certificate for the client and federation port? +* How do I tell Synapse to reload my keys/certificates after I replace them? + + +### Timeline + +**5th Feb 2019 - Synapse 0.99.0 is released.** + +All server admins are encouraged to upgrade. + +0.99.0: + +- provides support for ACME to make setting up Let's Encrypt certs easy, as + well as .well-known support. + +- does not enforce that a valid CA cert is present on the federation API, but + rather makes it easy to set one up. + +- provides support for .well-known + +Admins should upgrade and configure a valid CA cert. Homeservers that require a +.well-known entry (see below), should retain their SRV record and use it +alongside their .well-known record. + +**>= 5th March 2019 - Synapse 1.0.0 is released** + +1.0.0 will land no sooner than 1 month after 0.99.0, leaving server admins one +month after 5th February to upgrade to 0.99.0 and deploy their certificates. In +accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html) +1.0.0 will enforce federation checks. This means that any homeserver without a +valid certificate after this point will no longer be able to federate with +1.0.0 servers. + +### Synapse 0.99.0 has just been released, what do I need to do right now? + +Upgrade as soon as you can in preparation for Synapse 1.0.0. + +### How do I upgrade? + +Follow the upgrade notes here [UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst) + +### What will happen if I do not set up a valid federation certificate immediately? + +Nothing initially, but once 1.0.0 is in the wild it will not be possible to +federate with 1.0.0 servers. + +### What will happen if I do nothing at all? + +If the admin takes no action at all, and remains on a Synapse < 0.99.0 then the +homeserver will be unable to federate with those who have implemented +.well-known. Then, as above, once the month upgrade window has expired the +homeserver will not be able to federate with any Synapse >= 1.0.0 + +### When do I need a SRV record or .well-known URI? + +If your homeserver listens on the default federation port (8448), and your +server_name points to the host that your homeserver runs on, you do not need an +SRV record or .well-known/matrix/server URI.\ +For instance, if you registered example.com and pointed its DNS A record at a +fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host, +giving it a server_name of example.com, and it would automatically generate a +valid TLS certificate for you via Let's Encrypt and no SRV record or +.well-known URI would be needed. + +This is the common case, although you can add an SRV record or +.well-known/matrix/server URI for completeness if you wish. + +**However**, if your server does not listen on port 8448, or if your server_name +does not point to the host that your homeserver runs on, you will need to let +other servers know how to find it. + +The easiest way to do this is with a .well-known/matrix/server URI on the +webroot of the domain to advertise your server. For instance, if you ran +"matrixhosting.com" and you were hosting a Matrix server for example.com, you +would ask example.com to create a file at: + +`https://example.com/.well-known/matrix/server` + +with contents: + +`{"m.server": "example.matrixhosting.com:8448"}` + +...which would tell servers trying to connect to example.com to instead connect +to example.matrixhosting.com on port 8448. You would then configure Synapse +with a server_name of "example.com", but generate a TLS certificate for +example.matrixhosting.com. + +As an alternative, you can still use an SRV DNS record for the delegation, but +this will require you to have a certificate for the matrix domain (example.com +in this example). See "Can I still use an SRV record?". + +### Can I still use an SRV record? + +Firstly, if you didn't need an SRV record before (because your server is +listening on port 8448 of your server_name), you certainly don't need one now: +the defaults are still the same. + +If you previously had an SRV record, you can keep using it provided you are +able to give Synapse a TLS certificate corresponding to your server name. For +example, suppose you had the following SRV record, which directs matrix traffic +for example.com to matrix.example.com:443: + +``` +_matrix._tcp.example.com. IN SRV 10 5 443 matrix.example.com +``` + +In this case, Synapse must be given a certificate for example.com - or be +configured to acquire one from Let's Encrypt. + +If you are unable to give Synapse a certificate for your server_name, you will +also need to use a .well-known URI instead. However, see also "I have created a +.well-known URI. Do I still need an SRV record?". + +### I have created a .well-known URI. Do I still need an SRV record? + +As of Synapse 0.99, Synapse will first check for the existence of a .well-known +URL and follow any delegation it suggests. It will only then check for the +existence of an SRV record. + +That means that the SRV record will often be redundant. However, you should +remember that there may still be older versions of Synapse in the federation +which do not understand .well-known URIs, so if you removed your SRV record you +would no longer be able to federate with them. + +It is therefore best to leave the SRV record in place for now. Synapse 0.34 and +earlier will follow the SRV record (and not care about the invalid +certificate). Synapse 0.99 and later will follow the .well-known URI, with the +correct certificate chain. + +### It used to work just fine, why are you breaking everything? + +We have always wanted Matrix servers to be as easy to set up as possible, and +so back when we started federation in 2014 we didn't want admins to have to go +through the cumbersome process of buying a valid TLS certificate to run a +server. This was before Let's Encrypt came along and made getting a free and +valid TLS certificate straightforward. So instead, we adopted a system based on +[Perspectives](https://en.wikipedia.org/wiki/Convergence_(SSL)): an approach +where you check a set of "notary servers" (in practice, homeservers) to vouch +for the validity of a certificate rather than having it signed by a CA. As long +as enough different notaries agree on the certificate's validity, then it is +trusted. + +However, in practice this has never worked properly. Most people only use the +default notary server (matrix.org), leading to inadvertent centralisation which +we want to eliminate. Meanwhile, we never implemented the full consensus +algorithm to query the servers participating in a room to determine consensus +on whether a given certificate is valid. This is fiddly to get right +(especially in face of sybil attacks), and we found ourselves questioning +whether it was worth the effort to finish the work and commit to maintaining a +secure certificate validation system as opposed to focusing on core Matrix +development. + +Meanwhile, Let's Encrypt came along in 2016, and put the final nail in the +coffin of the Perspectives project (which was already pretty dead). So, the +Spec Core Team decided that a better approach would be to mandate valid TLS +certificates for federation alongside the rest of the Web. More details can be +found in +[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach). + +This results in a breaking change, which is disruptive, but absolutely critical +for the security model. However, the existence of Let's Encrypt as a trivial +way to replace the old self-signed certificates with valid CA-signed ones helps +smooth things over massively, especially as Synapse can now automate Let's +Encrypt certificate generation if needed. + +### Can I manage my own certificates rather than having Synapse renew certificates itself? + +Yes, you are welcome to manage your certificates yourself. Synapse will only +attempt to obtain certificates from Let's Encrypt if you configure it to do +so.The only requirement is that there is a valid TLS cert present for +federation end points. + +### Do you still recommend against using a reverse-proxy on the federation port? + +We no longer actively recommend against using a reverse proxy. Many admins will +find it easier to direct federation traffic to a reverse-proxy and manage their +own TLS certificates, and this is a supported configuration. + +### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? + +Practically speaking, this is no longer necessary. + +If you are using a reverse-proxy for all of your TLS traffic, then you can set +`no_tls: True`. In that case, the only reason Synapse needs the certificate is +to populate a legacy 'tls_fingerprints' field in the federation API. This is +ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will +check it is when attempting to fetch the server keys - and generally this is +delegated via `matrix.org`, which is on 0.99.0. + +However, there is a bug in Synapse 0.99.0 +[4554]() which prevents +Synapse from starting if you do not give it a TLS certificate. To work around +this, you can give it any TLS certificate at all. This will be fixed soon. + +### Do I need the same certificate for the client and federation port? + +No. There is nothing stopping you doing so, particularly if you are using a +reverse-proxy. However, Synapse will use the same certificate on any ports +where TLS is configured. + +### How do I tell Synapse to reload my keys/certificates after I replace them? + +Synapse will reload the keys and certificates when it receives a SIGHUP - for +example kill -HUP $(cat homeserver.pid). Alternatively, simply restart Synapse, +though this will result in downtime while it restarts. -- cgit 1.4.1 From 39bf0ea2e860948adcd47f0f9cf8ff73b8abf841 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 5 Feb 2019 18:11:26 +0000 Subject: Add notes on SRV and .well-known (#4573) --- docs/MSC1711_certificates_FAQ.md | 158 +++++++++++++++++++++++++++++---------- 1 file changed, 117 insertions(+), 41 deletions(-) (limited to 'docs') diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index 0dcef57733..efe6330647 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -19,19 +19,9 @@ certificate.** Admins will have 1 month to do so, after which 1.0.0 will be released and those servers without a valid certificate will not longer be able to federate with >= 1.0.0 servers. -If you are unable to generate a valid TLS certificate for your server (e.g. -because you run it on behalf of someone who doesn't want to give you a TLS -certificate for their domain, or simply because the matrix domain is hosted on -a different server), then you can now create a /.well-known/matrix/server file -on the matrix domain in order to delegate Matrix hosting to another domain. - Admins who currently use SRV records to delegate a domain **which they do not -control TLS for** will need to switch to using .well-known/matrix/server - though -they should retain their SRV record while the federation upgrades over the -course of the month.  Other SRV records are unaffected. - -Full upgrade notes can be found in -[UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst). -What follows is a timeline and some frequently asked questions. +Full details on how to carry out this configuration change is given +[below](#configuring-certificates-for-compatibility-with-synapse-100). A +timeline and some frequently asked questions are also given below. For more details and context on the release of the r0.1 Server/Server API and imminent Matrix 1.0 release, you can also see our @@ -39,25 +29,26 @@ imminent Matrix 1.0 release, you can also see our ## Contents * Timeline -* Synapse 0.99.0 has just been released, what do I need to do right now? -* How do I upgrade? -* What will happen if I do not set up a valid federation certificate - immediately? -* What will happen if I do nothing at all? -* When do I need a SRV record or .well-known URI? -* Can I still use an SRV record? -* I have created a .well-known URI. Do I still need an SRV record? -* It used to work just fine, why are you breaking everything? -* Can I manage my own certificates rather than having Synapse renew - certificates itself? -* Do you still recommend against using a reverse-proxy on the federation port? -* Do I still need to give my TLS certificates to Synapse if I am using a - reverse-proxy? -* Do I need the same certificate for the client and federation port? -* How do I tell Synapse to reload my keys/certificates after I replace them? - - -### Timeline +* Configuring certificates for compatibility with Synapse 1.0 +* FAQ + * Synapse 0.99.0 has just been released, what do I need to do right now? + * How do I upgrade? + * What will happen if I do not set up a valid federation certificate + immediately? + * What will happen if I do nothing at all? + * When do I need a SRV record or .well-known URI? + * Can I still use an SRV record? + * I have created a .well-known URI. Do I still need an SRV record? + * It used to work just fine, why are you breaking everything? + * Can I manage my own certificates rather than having Synapse renew + certificates itself? + * Do you still recommend against using a reverse-proxy on the federation port? + * Do I still need to give my TLS certificates to Synapse if I am using a + reverse-proxy? + * Do I need the same certificate for the client and federation port? + * How do I tell Synapse to reload my keys/certificates after I replace them? + +## Timeline **5th Feb 2019 - Synapse 0.99.0 is released.** @@ -82,10 +73,96 @@ alongside their .well-known record. 1.0.0 will land no sooner than 1 month after 0.99.0, leaving server admins one month after 5th February to upgrade to 0.99.0 and deploy their certificates. In accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html) -1.0.0 will enforce federation checks. This means that any homeserver without a +1.0.0 will enforce certificate validity. This means that any homeserver without a valid certificate after this point will no longer be able to federate with 1.0.0 servers. + +## Configuring certificates for compatibility with Synapse 1.0.0 + +### If you do not currently have an SRV record + +In this case, your `server_name` points to the host where your Synapse is +running. There is no need to create a `.well-known` URI or an SRV record, but +you will need to give Synapse a valid, signed, certificate. + +The easiest way to do that is with Synapse's built-in ACME (Let's Encrypt) +support. Full details are in [ACME.md](./ACME.md) but, in a nutshell: + + 1. Allow Synapse to listen on port 80 with `authbind`, or forward it from a + reverse proxy. + 2. Enable acme support in `homeserver.yaml`. + 3. Move your old certificates out of the way. + 4. Restart Synapse. + +### If you do have an SRV record currently + +If you are using an SRV record, your matrix domain (`server_name`) may not +point to the same host that your Synapse is running on (the 'target +domain'). (If it does, you can follow the recommendation above; otherwise, read +on.) + +Let's assume that your `server_name` is `example.com`, and your Synapse is +hosted at a target domain of `customer.example.net`. Currently you should have +an SRV record which looks like: + +``` +_matrix._tcp.example.com. IN SRV 10 5 443 customer.example.net. +``` + +In this situation, you have two choices for how to proceed: + +#### Option 1: give Synapse a certificate for your matrix domain + +Synapse 1.0 will expect your server to present a TLS certificate for your +`server_name` (`example.com` in the above example). You can achieve this by +doing one of the following: + + * Acquire a certificate for the `server_name` yourself (for example, using + `certbot`), and give it and the key to Synapse via `tls_certificate_path` + and `tls_private_key_path`, or: + + * Use Synapse's [ACME support](./ACME.md), and forward port 80 on the + `server_name` domain to your Synapse instance, or: + + * Set up a reverse-proxy on port 8448 on the `server_name` domain, which + forwards to Synapse. Once it is set up, you can remove the SRV record. + +#### Option 2: add a .well-known file to delegate your matrix traffic + +This will allow you to keep Synapse on a separate domain, without having to +give it a certificate for the matrix domain. + +You can do this with a `.well-known` file as follows: + + 1. Keep the SRV record in place - it is needed for backwards compatibility + with Synapse 0.34 and earlier. + + 2. Give synapse a certificate corresponding to the target domain + (`customer.example.net` in the above example). Currently Synapse's ACME + support [does not support + this](https://github.com/matrix-org/synapse/issues/4552), so you will have + to acquire a certificate yourself and give it to Synapse via + `tls_certificate_path` and `tls_private_key_path`. + + 3. Restart Synapse to ensure the new certificate is loaded. + + 4. Arrange for a `.well-known` file at + `https:///.well-known/matrix/server` with contents: + + ```json + {"m.server": ":"} + ``` + + In the above example, `https://example.com/.well-known/matrix/server` + should have the contents: + + ```json + {"m.server": "customer.example.net:443"} + ``` + +## FAQ + ### Synapse 0.99.0 has just been released, what do I need to do right now? Upgrade as soon as you can in preparation for Synapse 1.0.0. @@ -126,14 +203,13 @@ other servers know how to find it. The easiest way to do this is with a .well-known/matrix/server URI on the webroot of the domain to advertise your server. For instance, if you ran -"matrixhosting.com" and you were hosting a Matrix server for example.com, you -would ask example.com to create a file at: - -`https://example.com/.well-known/matrix/server` +"matrixhosting.com" and you were hosting a Matrix server for `example.com`, you +would ask `example.com` to create a file at +`https://example.com/.well-known/matrix/server` with contents: -with contents: - -`{"m.server": "example.matrixhosting.com:8448"}` +```json +{"m.server": "example.matrixhosting.com:8448"} +``` ...which would tell servers trying to connect to example.com to instead connect to example.matrixhosting.com on port 8448. You would then configure Synapse @@ -231,7 +307,7 @@ We no longer actively recommend against using a reverse proxy. Many admins will find it easier to direct federation traffic to a reverse-proxy and manage their own TLS certificates, and this is a supported configuration. -### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? +### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? Practically speaking, this is no longer necessary. -- cgit 1.4.1 From b05dd4ac06e88797477ef1991d1b2e44583afb9a Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 5 Feb 2019 18:59:57 +0000 Subject: faq cleanups --- UPGRADE.rst | 2 +- docs/MSC1711_certificates_FAQ.md | 54 ++++++++++++++-------------------------- 2 files changed, 19 insertions(+), 37 deletions(-) (limited to 'docs') diff --git a/UPGRADE.rst b/UPGRADE.rst index 3643477250..d869b7111b 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -57,7 +57,7 @@ will need to replace any self-signed certificates with those verified by a root CA. Information on how to do so can be found at `the ACME docs `_. -For more information on configuring TLS certificates see the `FAQ `_ +For more information on configuring TLS certificates see the `FAQ `_. Upgrading to v0.34.0 ==================== diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index efe6330647..eee37d9457 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -1,4 +1,4 @@ -# MSC 1711 Certificates FAQ +# MSC1711 Certificates FAQ The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It supports the r0.1 release of the server to server specification, but is @@ -165,11 +165,8 @@ You can do this with a `.well-known` file as follows: ### Synapse 0.99.0 has just been released, what do I need to do right now? -Upgrade as soon as you can in preparation for Synapse 1.0.0. - -### How do I upgrade? - -Follow the upgrade notes here [UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst) +Upgrade as soon as you can in preparation for Synapse 1.0.0, and update your +TLS certificates as [above](#configuring-certificates-for-compatibility-with-synapse-100). ### What will happen if I do not set up a valid federation certificate immediately? @@ -186,39 +183,24 @@ homeserver will not be able to federate with any Synapse >= 1.0.0 ### When do I need a SRV record or .well-known URI? If your homeserver listens on the default federation port (8448), and your -server_name points to the host that your homeserver runs on, you do not need an -SRV record or .well-known/matrix/server URI.\ -For instance, if you registered example.com and pointed its DNS A record at a +`server_name` points to the host that your homeserver runs on, you do not need an +SRV record or `.well-known/matrix/server` URI. + +For instance, if you registered `example.com` and pointed its DNS A record at a fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host, -giving it a server_name of example.com, and it would automatically generate a +giving it a server_name of `example.com`, and it would automatically generate a valid TLS certificate for you via Let's Encrypt and no SRV record or -.well-known URI would be needed. +`.well-known` URI would be needed. This is the common case, although you can add an SRV record or -.well-known/matrix/server URI for completeness if you wish. +`.well-known/matrix/server` URI for completeness if you wish. -**However**, if your server does not listen on port 8448, or if your server_name +**However**, if your server does not listen on port 8448, or if your `server_name` does not point to the host that your homeserver runs on, you will need to let other servers know how to find it. -The easiest way to do this is with a .well-known/matrix/server URI on the -webroot of the domain to advertise your server. For instance, if you ran -"matrixhosting.com" and you were hosting a Matrix server for `example.com`, you -would ask `example.com` to create a file at -`https://example.com/.well-known/matrix/server` with contents: - -```json -{"m.server": "example.matrixhosting.com:8448"} -``` - -...which would tell servers trying to connect to example.com to instead connect -to example.matrixhosting.com on port 8448. You would then configure Synapse -with a server_name of "example.com", but generate a TLS certificate for -example.matrixhosting.com. - -As an alternative, you can still use an SRV DNS record for the delegation, but -this will require you to have a certificate for the matrix domain (example.com -in this example). See "Can I still use an SRV record?". +In this case, you should see ["If you do have an SRV record +currently"](#if-you-do-have-an-srv-record-currently) above. ### Can I still use an SRV record? @@ -244,13 +226,13 @@ also need to use a .well-known URI instead. However, see also "I have created a ### I have created a .well-known URI. Do I still need an SRV record? -As of Synapse 0.99, Synapse will first check for the existence of a .well-known -URL and follow any delegation it suggests. It will only then check for the +As of Synapse 0.99, Synapse will first check for the existence of a `.well-known` +URI and follow any delegation it suggests. It will only then check for the existence of an SRV record. That means that the SRV record will often be redundant. However, you should remember that there may still be older versions of Synapse in the federation -which do not understand .well-known URIs, so if you removed your SRV record you +which do not understand `.well-known` URIs, so if you removed your SRV record you would no longer be able to federate with them. It is therefore best to leave the SRV record in place for now. Synapse 0.34 and @@ -332,5 +314,5 @@ where TLS is configured. ### How do I tell Synapse to reload my keys/certificates after I replace them? Synapse will reload the keys and certificates when it receives a SIGHUP - for -example kill -HUP $(cat homeserver.pid). Alternatively, simply restart Synapse, -though this will result in downtime while it restarts. +example `kill -HUP $(cat homeserver.pid)`. Alternatively, simply restart +Synapse, though this will result in downtime while it restarts. -- cgit 1.4.1 From 9b7aa543d9fc90d3a754ab129a619cfff6a71355 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Thu, 7 Feb 2019 18:46:02 +0000 Subject: clarify option 1 --- docs/MSC1711_certificates_FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index eee37d9457..414af96ef3 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -112,7 +112,7 @@ _matrix._tcp.example.com. IN SRV 10 5 443 customer.example.net. In this situation, you have two choices for how to proceed: -#### Option 1: give Synapse a certificate for your matrix domain +#### Option 1: give Synapse (or a reverse-proxy) a certificate for your matrix domain Synapse 1.0 will expect your server to present a TLS certificate for your `server_name` (`example.com` in the above example). You can achieve this by -- cgit 1.4.1 From c17b128b837f58cb59ba75803e32c8a720cf8501 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Thu, 7 Feb 2019 19:18:08 +0000 Subject: Update ACME docs to include port instructions (#4578) --- changelog.d/4578.misc | 1 + docs/ACME.md | 26 +++++++++++++++----------- 2 files changed, 16 insertions(+), 11 deletions(-) create mode 100644 changelog.d/4578.misc (limited to 'docs') diff --git a/changelog.d/4578.misc b/changelog.d/4578.misc new file mode 100644 index 0000000000..d1c006bb6b --- /dev/null +++ b/changelog.d/4578.misc @@ -0,0 +1 @@ +Add port configuration information to ACME instructions. \ No newline at end of file diff --git a/docs/ACME.md b/docs/ACME.md index 8fb2bd66a9..e555c7c939 100644 --- a/docs/ACME.md +++ b/docs/ACME.md @@ -41,10 +41,10 @@ placed in Synapse's config directory without the need for any ACME setup. The main steps for enabling ACME support in short summary are: -1. Allow Synapse to listen on port 80 with authbind, or forward it from a reverse-proxy. -1. Set `acme:enabled` to `true` in homeserver.yaml. +1. Allow Synapse to listen for incoming ACME challenges. +1. Enable ACME support in `homeserver.yaml`. 1. Move your old certificates (files `example.com.tls.crt` and `example.com.tls.key` out of the way if they currently exist at the paths specified in `homeserver.yaml`. -1. Restart Synapse +1. Restart Synapse. Detailed instructions for each step are provided below. @@ -71,7 +71,7 @@ location /.well-known/acme-challenge { } ``` -For Apache, add the following to your existing webserver config:: +For Apache, add the following to your existing webserver config: ``` ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge @@ -79,6 +79,14 @@ ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-cha Make sure to restart/reload your webserver after making changes. +Now make the relevant changes in `homeserver.yaml` to enable ACME support: + +``` +acme: + enabled: true + port: 8009 +``` + #### Authbind @@ -102,24 +110,20 @@ sudo touch /etc/authbind/byport/80 sudo chmod 777 /etc/authbind/byport/80 ``` -When Synapse is started, use the following syntax:: +When Synapse is started, use the following syntax: ``` authbind --deep ``` -### Config file editing - -Once Synapse is able to listen on port 80 for ACME challenge -requests, it must be told to perform ACME provisioning by setting `enabled` -to true under the `acme` section in `homeserver.yaml`: +Make the relevant changes in `homeserver.yaml` to enable ACME support: ``` acme: enabled: true ``` -### Starting synapse +### (Re)starting synapse Ensure that the certificate paths specified in `homeserver.yaml` (`tls_certificate_path` and `tls_private_key_path`) do not currently point to any files. Synapse will not provision certificates if files exist, as it does not want to overwrite existing certificates. -- cgit 1.4.1 From 9285d5c2ce897cf71bff42eca2cfd59e04e1b056 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 Feb 2019 17:49:53 +0000 Subject: Update MSC1711 FAQ to be explicit about well-known A surprising number of people are using the well-known method, and are simply copying the example configuration. This is problematic as the example includes an explicit port, which causes inbound federation requests to have the HTTP Host header include the port, upsetting some reverse proxies. Given that, we update the well-known example to be more explicit about the various ways you can set it up, and the consequence of using an explict port. --- docs/MSC1711_certificates_FAQ.md | 38 +++++++++++++++++++++++++++----------- 1 file changed, 27 insertions(+), 11 deletions(-) (limited to 'docs') diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index eee37d9457..a3a36d222e 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -107,10 +107,10 @@ hosted at a target domain of `customer.example.net`. Currently you should have an SRV record which looks like: ``` -_matrix._tcp.example.com. IN SRV 10 5 443 customer.example.net. +_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net. ``` -In this situation, you have two choices for how to proceed: +In this situation, you have three choices for how to proceed: #### Option 1: give Synapse a certificate for your matrix domain @@ -125,10 +125,16 @@ doing one of the following: * Use Synapse's [ACME support](./ACME.md), and forward port 80 on the `server_name` domain to your Synapse instance, or: - * Set up a reverse-proxy on port 8448 on the `server_name` domain, which - forwards to Synapse. Once it is set up, you can remove the SRV record. -#### Option 2: add a .well-known file to delegate your matrix traffic +### Option 2: run Synapse behind a reverse proxy + +If you have an existing reverse proxy set up with correct TLS certificates for +your domain, you can simply route all traffic through the reverse proxy by +updating the SRV record appropriately (or removing it, if the proxy listens on +8448). + + +#### Option 3: add a .well-known file to delegate your matrix traffic This will allow you to keep Synapse on a separate domain, without having to give it a certificate for the matrix domain. @@ -151,15 +157,25 @@ You can do this with a `.well-known` file as follows: `https:///.well-known/matrix/server` with contents: ```json - {"m.server": ":"} + {"m.server": ""} ``` - In the above example, `https://example.com/.well-known/matrix/server` - should have the contents: + where the target server name is resolved as usual (i.e. SRV lookup, falling + back to talking to port 8448). + + In the above example, where synapse is listening on port 8000, + `https://example.com/.well-known/matrix/server` should have `m.server` set to one of: + + 1. `customer.example.net` ─ with a SRV record on + `_matrix._tcp.customer.example.com` pointing to port 8000, or: + + 2. `customer.example.net` ─ updating synapse to listen on the default port + 8448, or: + + 3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy + on `customer.example.net:8000` it correctly handles HTTP requests with + Host header set to `customer.example.net:8000`. - ```json - {"m.server": "customer.example.net:443"} - ``` ## FAQ -- cgit 1.4.1 From 7cadc4c918c207a574ea15bd1e3793d8a48b7beb Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 7 Feb 2019 19:29:20 +0000 Subject: cleanups --- docs/MSC1711_certificates_FAQ.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) (limited to 'docs') diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index 579c5dffce..0a781d00e3 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -112,7 +112,7 @@ _matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net. In this situation, you have three choices for how to proceed: -#### Option 1: give Synapse (or a reverse-proxy) a certificate for your matrix domain +#### Option 1: give Synapse a certificate for your matrix domain Synapse 1.0 will expect your server to present a TLS certificate for your `server_name` (`example.com` in the above example). You can achieve this by @@ -123,8 +123,7 @@ doing one of the following: and `tls_private_key_path`, or: * Use Synapse's [ACME support](./ACME.md), and forward port 80 on the - `server_name` domain to your Synapse instance, or: - + `server_name` domain to your Synapse instance. ### Option 2: run Synapse behind a reverse proxy @@ -133,7 +132,6 @@ your domain, you can simply route all traffic through the reverse proxy by updating the SRV record appropriately (or removing it, if the proxy listens on 8448). - #### Option 3: add a .well-known file to delegate your matrix traffic This will allow you to keep Synapse on a separate domain, without having to -- cgit 1.4.1 From acb2ac5863d34e2d01fa53598aecda274dc2fb26 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 Feb 2019 19:30:32 +0000 Subject: Update MSC1711 FAQ to be explicit about well-known (#4584) A surprising number of people are using the well-known method, and are simply copying the example configuration. This is problematic as the example includes an explicit port, which causes inbound federation requests to have the HTTP Host header include the port, upsetting some reverse proxies. Given that, we update the well-known example to be more explicit about the various ways you can set it up, and the consequence of using an explict port. --- changelog.d/4584.misc | 1 + docs/MSC1711_certificates_FAQ.md | 40 +++++++++++++++++++++++++++------------- 2 files changed, 28 insertions(+), 13 deletions(-) create mode 100644 changelog.d/4584.misc (limited to 'docs') diff --git a/changelog.d/4584.misc b/changelog.d/4584.misc new file mode 100644 index 0000000000..4dec2e2b5c --- /dev/null +++ b/changelog.d/4584.misc @@ -0,0 +1 @@ +Update MSC1711 FAQ to calrify .well-known usage diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index 414af96ef3..0a781d00e3 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -107,12 +107,12 @@ hosted at a target domain of `customer.example.net`. Currently you should have an SRV record which looks like: ``` -_matrix._tcp.example.com. IN SRV 10 5 443 customer.example.net. +_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net. ``` -In this situation, you have two choices for how to proceed: +In this situation, you have three choices for how to proceed: -#### Option 1: give Synapse (or a reverse-proxy) a certificate for your matrix domain +#### Option 1: give Synapse a certificate for your matrix domain Synapse 1.0 will expect your server to present a TLS certificate for your `server_name` (`example.com` in the above example). You can achieve this by @@ -123,12 +123,16 @@ doing one of the following: and `tls_private_key_path`, or: * Use Synapse's [ACME support](./ACME.md), and forward port 80 on the - `server_name` domain to your Synapse instance, or: + `server_name` domain to your Synapse instance. - * Set up a reverse-proxy on port 8448 on the `server_name` domain, which - forwards to Synapse. Once it is set up, you can remove the SRV record. +### Option 2: run Synapse behind a reverse proxy -#### Option 2: add a .well-known file to delegate your matrix traffic +If you have an existing reverse proxy set up with correct TLS certificates for +your domain, you can simply route all traffic through the reverse proxy by +updating the SRV record appropriately (or removing it, if the proxy listens on +8448). + +#### Option 3: add a .well-known file to delegate your matrix traffic This will allow you to keep Synapse on a separate domain, without having to give it a certificate for the matrix domain. @@ -151,15 +155,25 @@ You can do this with a `.well-known` file as follows: `https:///.well-known/matrix/server` with contents: ```json - {"m.server": ":"} + {"m.server": ""} ``` - In the above example, `https://example.com/.well-known/matrix/server` - should have the contents: + where the target server name is resolved as usual (i.e. SRV lookup, falling + back to talking to port 8448). + + In the above example, where synapse is listening on port 8000, + `https://example.com/.well-known/matrix/server` should have `m.server` set to one of: + + 1. `customer.example.net` ─ with a SRV record on + `_matrix._tcp.customer.example.com` pointing to port 8000, or: + + 2. `customer.example.net` ─ updating synapse to listen on the default port + 8448, or: + + 3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy + on `customer.example.net:8000` it correctly handles HTTP requests with + Host header set to `customer.example.net:8000`. - ```json - {"m.server": "customer.example.net:443"} - ``` ## FAQ -- cgit 1.4.1 From 4588b0d64a954ed4fef1f9034dec9e8407867a5c Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Fri, 8 Feb 2019 09:37:16 +0000 Subject: Update MSC1711_certificates_FAQ.md Fix incorrect heading level --- docs/MSC1711_certificates_FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index 0a781d00e3..c4a57d6ed1 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -125,7 +125,7 @@ doing one of the following: * Use Synapse's [ACME support](./ACME.md), and forward port 80 on the `server_name` domain to your Synapse instance. -### Option 2: run Synapse behind a reverse proxy +#### Option 2: run Synapse behind a reverse proxy If you have an existing reverse proxy set up with correct TLS certificates for your domain, you can simply route all traffic through the reverse proxy by -- cgit 1.4.1 From c475275926aeee906b76621444468280d5bf569b Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Mon, 11 Feb 2019 11:44:28 +0000 Subject: Clarifications for reverse proxy docs (#4607) Factor out the reverse proxy info to a separate file, add some more info on reverse-proxying the federation port. --- INSTALL.md | 5 ++- README.rst | 52 ++-------------------- changelog.d/4607.misc | 1 + docs/MSC1711_certificates_FAQ.md | 22 ++++++---- docs/reverse_proxy.rst | 94 ++++++++++++++++++++++++++++++++++++++++ docs/workers.rst | 5 +-- 6 files changed, 117 insertions(+), 62 deletions(-) create mode 100644 changelog.d/4607.misc create mode 100644 docs/reverse_proxy.rst (limited to 'docs') diff --git a/INSTALL.md b/INSTALL.md index cbe4bda120..e496a13b21 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -359,8 +359,9 @@ Synapse v1.0. Instructions for having Synapse automatically provision and renew If you would like to use your own certificates, you can do so by changing `tls_certificate_path` and `tls_private_key_path` in `homeserver.yaml`; -alternatively, you can use a reverse-proxy. Apart from port 8448 using TLS, -both ports are the same in the default configuration. +alternatively, you can use a reverse proxy. See +[docs/reverse_proxy.rst](docs/reverse_proxy.rst) for information on configuring +a reverse proxy. ## Registering a user diff --git a/README.rst b/README.rst index e666b3b427..bc7cb5f784 100644 --- a/README.rst +++ b/README.rst @@ -263,6 +263,8 @@ So, things to check are: (it should be ``_matrix._tcp.``), and that the port and hostname it specifies are reachable from outside your network. +.. TODO: add a note about forgetting ``nocanon`` on a reverse-proxy config + Running a Demo Federation of Synapses ------------------------------------- @@ -290,7 +292,6 @@ The advantages of Postgres include: For information on how to install and use PostgreSQL, please see `docs/postgres.rst `_. - .. _reverse-proxy: Using a reverse proxy with Synapse @@ -304,54 +305,7 @@ It is recommended to put a reverse proxy such as doing so is that it means that you can expose the default https port (443) to Matrix clients without needing to run Synapse with root privileges. -The most important thing to know here is that Matrix clients and other Matrix -servers do not necessarily need to connect to your server via the same -port. Indeed, clients will use port 443 by default, whereas servers default to -port 8448. Where these are different, we refer to the 'client port' and the -'federation port'. - -All Matrix endpoints begin with ``/_matrix``, so an example nginx -configuration for forwarding client connections to Synapse might look like:: - - server { - listen 443 ssl; - listen [::]:443 ssl; - server_name matrix.example.com; - - location /_matrix { - proxy_pass http://localhost:8008; - proxy_set_header X-Forwarded-For $remote_addr; - } - } - -an example Caddy configuration might look like:: - - matrix.example.com { - proxy /_matrix http://localhost:8008 { - transparent - } - } - -and an example Apache configuration might look like:: - - - SSLEngine on - ServerName matrix.example.com; - - - ProxyPass http://127.0.0.1:8008/_matrix nocanon - ProxyPassReverse http://127.0.0.1:8008/_matrix - - - -You will also want to set ``bind_addresses: ['127.0.0.1']`` and ``x_forwarded: true`` -for port 8008 in ``homeserver.yaml`` to ensure that client IP addresses are -recorded correctly. - -Having done so, you can then use ``https://matrix.example.com`` (instead of -``https://matrix.example.com:8448``) as the "Custom server" when `Connecting to -Synapse from a client`_. - +For information on configuring one, see ``_. Identity Servers ================ diff --git a/changelog.d/4607.misc b/changelog.d/4607.misc new file mode 100644 index 0000000000..160a824378 --- /dev/null +++ b/changelog.d/4607.misc @@ -0,0 +1 @@ +Clarifications for reverse proxy docs diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md index 0a781d00e3..2c52b0d517 100644 --- a/docs/MSC1711_certificates_FAQ.md +++ b/docs/MSC1711_certificates_FAQ.md @@ -42,9 +42,9 @@ imminent Matrix 1.0 release, you can also see our * It used to work just fine, why are you breaking everything? * Can I manage my own certificates rather than having Synapse renew certificates itself? - * Do you still recommend against using a reverse-proxy on the federation port? + * Do you still recommend against using a reverse proxy on the federation port? * Do I still need to give my TLS certificates to Synapse if I am using a - reverse-proxy? + reverse proxy? * Do I need the same certificate for the client and federation port? * How do I tell Synapse to reload my keys/certificates after I replace them? @@ -132,6 +132,9 @@ your domain, you can simply route all traffic through the reverse proxy by updating the SRV record appropriately (or removing it, if the proxy listens on 8448). +See [reverse_proxy.rst](reverse_proxy.rst) for information on setting up a +reverse proxy. + #### Option 3: add a .well-known file to delegate your matrix traffic This will allow you to keep Synapse on a separate domain, without having to @@ -297,17 +300,20 @@ attempt to obtain certificates from Let's Encrypt if you configure it to do so.The only requirement is that there is a valid TLS cert present for federation end points. -### Do you still recommend against using a reverse-proxy on the federation port? +### Do you still recommend against using a reverse proxy on the federation port? We no longer actively recommend against using a reverse proxy. Many admins will -find it easier to direct federation traffic to a reverse-proxy and manage their +find it easier to direct federation traffic to a reverse proxy and manage their own TLS certificates, and this is a supported configuration. +See [reverse_proxy.rst](reverse_proxy.rst) for information on setting up a +reverse proxy. + ### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? Practically speaking, this is no longer necessary. -If you are using a reverse-proxy for all of your TLS traffic, then you can set +If you are using a reverse proxy for all of your TLS traffic, then you can set `no_tls: True`. In that case, the only reason Synapse needs the certificate is to populate a legacy 'tls_fingerprints' field in the federation API. This is ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will @@ -321,9 +327,9 @@ this, you can give it any TLS certificate at all. This will be fixed soon. ### Do I need the same certificate for the client and federation port? -No. There is nothing stopping you doing so, particularly if you are using a -reverse-proxy. However, Synapse will use the same certificate on any ports -where TLS is configured. +No. There is nothing stopping you from using different certificates, +particularly if you are using a reverse proxy. However, Synapse will use the +same certificate on any ports where TLS is configured. ### How do I tell Synapse to reload my keys/certificates after I replace them? diff --git a/docs/reverse_proxy.rst b/docs/reverse_proxy.rst new file mode 100644 index 0000000000..d8aaac8a08 --- /dev/null +++ b/docs/reverse_proxy.rst @@ -0,0 +1,94 @@ +Using a reverse proxy with Synapse +================================== + +It is recommended to put a reverse proxy such as +`nginx `_, +`Apache `_, +`Caddy `_ or +`HAProxy `_ in front of Synapse. One advantage of +doing so is that it means that you can expose the default https port (443) to +Matrix clients without needing to run Synapse with root privileges. + +**NOTE**: Your reverse proxy must not 'canonicalise' or 'normalise' the +requested URI in any way (for example, by decoding ``%xx`` escapes). Beware +that Apache *will* canonicalise URIs unless you specifify ``nocanon``. + +When setting up a reverse proxy, remember that Matrix clients and other Matrix +servers do not necessarily need to connect to your server via the same server +name or port. Indeed, clients will use port 443 by default, whereas servers +default to port 8448. Where these are different, we refer to the 'client port' +and the 'federation port'. See `Setting up federation +<../README.rst#setting-up-federation>`_ for more details of the algorithm used for +federation connections. + +Let's assume that we expect clients to connect to our server at +``https://matrix.example.com``, and other servers to connect at +``https://example.com:8448``. Here are some example configurations: + +* nginx:: + + server { + listen 443 ssl; + listen [::]:443 ssl; + server_name matrix.example.com; + + location /_matrix { + proxy_pass http://localhost:8008; + proxy_set_header X-Forwarded-For $remote_addr; + } + } + + server { + listen 8448 ssl default_server; + listen [::]:8448 ssl default_server; + server_name example.com; + + location / { + proxy_pass http://localhost:8008; + proxy_set_header X-Forwarded-For $remote_addr; + } + } + +* Caddy:: + + matrix.example.com { + proxy /_matrix http://localhost:8008 { + transparent + } + } + + example.com:8448 { + proxy / http://localhost:8008 { + transparent + } + } + +* Apache (note the ``nocanon`` options here!):: + + + SSLEngine on + ServerName matrix.example.com; + + + ProxyPass http://127.0.0.1:8008/_matrix nocanon + ProxyPassReverse http://127.0.0.1:8008/_matrix + + + + + SSLEngine on + ServerName example.com; + + + ProxyPass http://127.0.0.1:8008/_matrix nocanon + ProxyPassReverse http://127.0.0.1:8008/_matrix + + + +You will also want to set ``bind_addresses: ['127.0.0.1']`` and ``x_forwarded: true`` +for port 8008 in ``homeserver.yaml`` to ensure that client IP addresses are +recorded correctly. + +Having done so, you can then use ``https://matrix.example.com`` (instead of +``https://matrix.example.com:8448``) as the "Custom server" when connecting to +Synapse from a client. diff --git a/docs/workers.rst b/docs/workers.rst index 101e950020..dd3a84ba0d 100644 --- a/docs/workers.rst +++ b/docs/workers.rst @@ -26,9 +26,8 @@ Configuration To make effective use of the workers, you will need to configure an HTTP reverse-proxy such as nginx or haproxy, which will direct incoming requests to the correct worker, or to the main synapse instance. Note that this includes -requests made to the federation port. The caveats regarding running a -reverse-proxy on the federation port still apply (see -https://github.com/matrix-org/synapse/blob/master/README.rst#reverse-proxying-the-federation-port). +requests made to the federation port. See ``_ for +information on setting up a reverse proxy. To enable workers, you need to add two replication listeners to the master synapse, e.g.:: -- cgit 1.4.1 From 8b9ae6d3a665e861072ee92bba38dd44a13d9f54 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 18 Feb 2019 15:25:44 +0000 Subject: Update docs --- docs/workers.rst | 6 ++++++ synapse/rest/client/v2_alpha/register.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/workers.rst b/docs/workers.rst index dd3a84ba0d..6ce7d88c11 100644 --- a/docs/workers.rst +++ b/docs/workers.rst @@ -223,6 +223,12 @@ following regular expressions:: ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ +Additionally, the following REST endpoints can be handled, but all requests must +be routed to the same instance:: + + ^/_matrix/client/(api/v1|r0|unstable)/register$ + + ``synapse.app.user_dir`` ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py index d78da50787..c52280c50c 100644 --- a/synapse/rest/client/v2_alpha/register.py +++ b/synapse/rest/client/v2_alpha/register.py @@ -665,7 +665,7 @@ class RegisterRestServlet(RestServlet): device. is_guest (bool): Whether this is a guest account Returns: - defer.Deferred[(str, str)]: Tuple of device ID and access token + defer.Deferred[tuple[str, str]]: Tuple of device ID and access token """ if self.hs.config.worker_app: r = yield self._register_device_client( -- cgit 1.4.1 From 128902d60a79b3fc36be084413e742ecf9300c82 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 18 Feb 2019 17:21:51 +0000 Subject: Update worker docs --- changelog.d/4666.feature | 2 +- docs/workers.rst | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/changelog.d/4666.feature b/changelog.d/4666.feature index 421060f9f9..b3a3915eb0 100644 --- a/changelog.d/4666.feature +++ b/changelog.d/4666.feature @@ -1 +1 @@ -Allow registration to be handled by a worker instance. +Allow registration and login to be handled by a worker instance. diff --git a/docs/workers.rst b/docs/workers.rst index 6ce7d88c11..3ba5879f76 100644 --- a/docs/workers.rst +++ b/docs/workers.rst @@ -222,11 +222,12 @@ following regular expressions:: ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ + ^/_matrix/client/(api/v1|r0|unstable)/login$ Additionally, the following REST endpoints can be handled, but all requests must be routed to the same instance:: - ^/_matrix/client/(api/v1|r0|unstable)/register$ + ^/_matrix/client/(r0|unstable)/register$ ``synapse.app.user_dir`` -- cgit 1.4.1 From bc8fa1509d3885d111a2ef98e12e9ce66a19a3a8 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 19 Feb 2019 11:24:59 +0000 Subject: Documentation --- docs/tcp_replication.rst | 21 ++++++++++++++++++++- synapse/storage/_base.py | 8 ++++---- 2 files changed, 24 insertions(+), 5 deletions(-) (limited to 'docs') diff --git a/docs/tcp_replication.rst b/docs/tcp_replication.rst index 62225ba6f4..852f1113a3 100644 --- a/docs/tcp_replication.rst +++ b/docs/tcp_replication.rst @@ -137,7 +137,6 @@ for each stream so that on reconneciton it can start streaming from the correct place. Note: not all RDATA have valid tokens due to batching. See ``RdataCommand`` for more details. - Example ~~~~~~~ @@ -221,3 +220,23 @@ SYNC (S, C) See ``synapse/replication/tcp/commands.py`` for a detailed description and the format of each command. + + +Cache Invalidation Stream +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The cache invalidation stream is used to inform workers when they need to +invalidate any of their caches in the data store. This is done by streaming all +cache invalidations done on master down to the workers, assuming that any caches +on the workers also exist on the master. + +Each individual cache invalidation results in a row being sent down replication, +which includes the cache name (the name of the function) and they key to +invalidate. For example:: + + > RDATA caches 550953771 ["get_user_by_id", ["@bob:example.com"], 1550574873251] + +However, there are times when a number of caches need to be invalidated at the +same time with the same key. To reduce traffic we batch those invalidations into +a single poke by defining a special cache name that workers understand to mean +to expand to invalidate the correct caches. diff --git a/synapse/storage/_base.py b/synapse/storage/_base.py index f7c6d714ab..1c8d3f0026 100644 --- a/synapse/storage/_base.py +++ b/synapse/storage/_base.py @@ -1199,8 +1199,8 @@ class SQLBaseStore(object): Args: txn - room_id (str): Room were state changed - members_changed (set[str]): The user_ids of members that have changed + room_id (str): Room where state changed + members_changed (Iterable[str]): The user_ids of members that have changed """ txn.call_after(self._invalidate_state_caches, room_id, members_changed) @@ -1215,7 +1215,7 @@ class SQLBaseStore(object): not stream invalidations down replication. Args: - room_id (str): Room were state changed + room_id (str): Room where state changed members_changed (set[str]): The user_ids of members that have changed """ for member in members_changed: @@ -1237,7 +1237,7 @@ class SQLBaseStore(object): Args: txn cache_name (str) - keys (list[str]) + keys (iterable[str]) """ if isinstance(self.database_engine, PostgresEngine): -- cgit 1.4.1 From 62175a20e51b4ce71b9e7a18755a42e259bd2ff8 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 19 Feb 2019 11:38:40 +0000 Subject: Docs --- docs/tcp_replication.rst | 5 +++++ synapse/storage/_base.py | 5 +++-- 2 files changed, 8 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/docs/tcp_replication.rst b/docs/tcp_replication.rst index 852f1113a3..73436cea62 100644 --- a/docs/tcp_replication.rst +++ b/docs/tcp_replication.rst @@ -240,3 +240,8 @@ However, there are times when a number of caches need to be invalidated at the same time with the same key. To reduce traffic we batch those invalidations into a single poke by defining a special cache name that workers understand to mean to expand to invalidate the correct caches. + +Currently the special cache names are declared in ``synapse/storage/_base.py`` +and are: + +1. ``cs_cache_fake`` ─ invalidates caches that depend on the current state diff --git a/synapse/storage/_base.py b/synapse/storage/_base.py index 9db594bc42..f1a5366b95 100644 --- a/synapse/storage/_base.py +++ b/synapse/storage/_base.py @@ -1201,7 +1201,7 @@ class SQLBaseStore(object): Args: txn room_id (str): Room where state changed - members_changed (Iterable[str]): The user_ids of members that have changed + members_changed (iterable[str]): The user_ids of members that have changed """ txn.call_after(self._invalidate_state_caches, room_id, members_changed) @@ -1216,7 +1216,8 @@ class SQLBaseStore(object): Args: room_id (str): Room where state changed - members_changed (set[str]): The user_ids of members that have changed + members_changed (iterable[str]): The user_ids of members that have + changed """ for member in members_changed: self.get_rooms_for_user_with_stream_ordering.invalidate((member,)) -- cgit 1.4.1 From 16e0680498435d2a93b26f81f57bfa41c52c691b Mon Sep 17 00:00:00 2001 From: Benoît S Date: Thu, 21 Feb 2019 18:44:10 +0100 Subject: Added HAProxy example (#4660) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Added HAProxy example Proposal of an example with HAProxy. Asked by #4541. Signed-off-by: Benoît S. (“Benpro”) * Following suggestions of @richvdh --- changelog.d/4541.feature | 1 + docs/reverse_proxy.rst | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 changelog.d/4541.feature (limited to 'docs') diff --git a/changelog.d/4541.feature b/changelog.d/4541.feature new file mode 100644 index 0000000000..1d0e7bdfdc --- /dev/null +++ b/changelog.d/4541.feature @@ -0,0 +1 @@ +Added an HAProxy example in the reverse proxy documentation. Contributed by Benoît S. (“Benpro”). diff --git a/docs/reverse_proxy.rst b/docs/reverse_proxy.rst index d8aaac8a08..242935a62f 100644 --- a/docs/reverse_proxy.rst +++ b/docs/reverse_proxy.rst @@ -85,6 +85,24 @@ Let's assume that we expect clients to connect to our server at +* HAProxy:: + + frontend https + bind 0.0.0.0:443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1 + bind :::443 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1 + + # Matrix client traffic + acl matrix hdr(host) -i matrix.example.com + use_backend matrix if matrix + + frontend matrix-federation + bind 0.0.0.0:8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1 + bind :::8448 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1 + default_backend matrix + + backend matrix + server matrix 127.0.0.1:8008 + You will also want to set ``bind_addresses: ['127.0.0.1']`` and ``x_forwarded: true`` for port 8008 in ``homeserver.yaml`` to ensure that client IP addresses are recorded correctly. -- cgit 1.4.1 From fcd6f01dc790c1245b413e37d405a8d092397f98 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 22 Feb 2019 10:56:42 +0000 Subject: Minor tweaks to acme docs (#4689) --- changelog.d/4689.misc | 1 + docs/ACME.md | 19 +++++++++---------- 2 files changed, 10 insertions(+), 10 deletions(-) create mode 100644 changelog.d/4689.misc (limited to 'docs') diff --git a/changelog.d/4689.misc b/changelog.d/4689.misc new file mode 100644 index 0000000000..15c4d9404b --- /dev/null +++ b/changelog.d/4689.misc @@ -0,0 +1 @@ +Minor tweaks to acme docs. diff --git a/docs/ACME.md b/docs/ACME.md index e555c7c939..46136a9f2c 100644 --- a/docs/ACME.md +++ b/docs/ACME.md @@ -10,13 +10,14 @@ through [Let's Encrypt](https://letsencrypt.org/) if you tell it to. In the case that your `server_name` config variable is the same as the hostname that the client connects to, then the same certificate can be -used between client and federation ports without issue. +used between client and federation ports without issue. -For a sample configuration, please inspect the new ACME section in the example -generated config by running the `generate-config` executable. For example: +If your configuration file does not already have an `acme` section, you can +generate an example config by running the `generate_config` executable. For +example: ``` -~/synapse/env3/bin/generate-config +~/synapse/env3/bin/generate_config ``` You will need to provide Let's Encrypt (or another ACME provider) access to @@ -27,10 +28,9 @@ like `authbind` to allow Synapse to listen on port 80 without root access. (Do not run Synapse with root permissions!) Detailed instructions are available under "ACME setup" below. -If you are already using self-signed certificates, you will need to back up -or delete them (files `example.com.tls.crt` and `example.com.tls.key` in -Synapse's root directory), Synapse's ACME implementation will not overwrite -them. +If you already have certificates, you will need to back up or delete them +(files `example.com.tls.crt` and `example.com.tls.key` in Synapse's root +directory), Synapse's ACME implementation will not overwrite them. You may wish to use alternate methods such as Certbot to obtain a certificate from Let's Encrypt, depending on your server configuration. Of course, if you @@ -87,7 +87,6 @@ acme: port: 8009 ``` - #### Authbind `authbind` allows a program which does not run as root to bind to @@ -127,4 +126,4 @@ acme: Ensure that the certificate paths specified in `homeserver.yaml` (`tls_certificate_path` and `tls_private_key_path`) do not currently point to any files. Synapse will not provision certificates if files exist, as it does not want to overwrite existing certificates. -Finally, start/restart Synapse. \ No newline at end of file +Finally, start/restart Synapse. -- cgit 1.4.1