Merge tag 'v0.34.1rc1' into matrix-org-hotfixes

Synapse 0.34.1rc1 (2019-01-08)
==============================

Features
--------

- Special-case a support user for use in verifying behaviour of a given server. The support user does not appear in user directory or monthly active user counts. ([\#4141](https://github.com/matrix-org/synapse/issues/4141), [\#4344](https://github.com/matrix-org/synapse/issues/4344))
- Support for serving .well-known files ([\#4262](https://github.com/matrix-org/synapse/issues/4262))
- Rework SAML2 authentication ([\#4265](https://github.com/matrix-org/synapse/issues/4265), [\#4267](https://github.com/matrix-org/synapse/issues/4267))
- SAML2 authentication: Initialise user display name from SAML2 data ([\#4272](https://github.com/matrix-org/synapse/issues/4272))
- Synapse can now have its conditional/extra dependencies installed by pip. This functionality can be used by using `pip install matrix-synapse[feature]`, where feature is a comma separated list with the possible values `email.enable_notifs`, `matrix-synapse-ldap3`, `postgres`, `resources.consent`, `saml2`, `url_preview`, and `test`. If you want to install all optional dependencies, you can use "all" instead. ([\#4298](https://github.com/matrix-org/synapse/issues/4298), [\#4325](https://github.com/matrix-org/synapse/issues/4325), [\#4327](https://github.com/matrix-org/synapse/issues/4327))
- Add routes for reading account data. ([\#4303](https://github.com/matrix-org/synapse/issues/4303))
- Add opt-in support for v2 rooms ([\#4307](https://github.com/matrix-org/synapse/issues/4307))
- Add a script to generate a clean config file ([\#4315](https://github.com/matrix-org/synapse/issues/4315))
- Return server data in /login response ([\#4319](https://github.com/matrix-org/synapse/issues/4319))

Bugfixes
--------

- Fix contains_url check to be consistent with other instances in code-base and check that value is an instance of string. ([\#3405](https://github.com/matrix-org/synapse/issues/3405))
- Fix CAS login when username is not valid in an MXID ([\#4264](https://github.com/matrix-org/synapse/issues/4264))
- Send CORS headers for /media/config ([\#4279](https://github.com/matrix-org/synapse/issues/4279))
- Add 'sandbox' to CSP for media reprository ([\#4284](https://github.com/matrix-org/synapse/issues/4284))
- Make the new landing page prettier. ([\#4294](https://github.com/matrix-org/synapse/issues/4294))
- Fix deleting E2E room keys when using old SQLite versions. ([\#4295](https://github.com/matrix-org/synapse/issues/4295))
- The metric synapse_admin_mau:current previously did not update when config.mau_stats_only was set to True ([\#4305](https://github.com/matrix-org/synapse/issues/4305))
- Fixed per-room account data filters ([\#4309](https://github.com/matrix-org/synapse/issues/4309))
- Fix indentation in default config ([\#4313](https://github.com/matrix-org/synapse/issues/4313))
- Fix synapse:latest docker upload ([\#4316](https://github.com/matrix-org/synapse/issues/4316))
- Fix test_metric.py compatibility with prometheus_client 0.5. Contributed by Maarten de Vries <maarten@de-vri.es>. ([\#4317](https://github.com/matrix-org/synapse/issues/4317))
- Avoid packaging _trial_temp directory in -py3 debian packages ([\#4326](https://github.com/matrix-org/synapse/issues/4326))
- Check jinja version for consent resource ([\#4327](https://github.com/matrix-org/synapse/issues/4327))
- fix NPE in /messages by checking if all events were filtered out ([\#4330](https://github.com/matrix-org/synapse/issues/4330))
- Fix `python -m synapse.config` on Python 3. ([\#4356](https://github.com/matrix-org/synapse/issues/4356))

Deprecations and Removals
-------------------------

- Remove the deprecated v1/register API on Python 2. It was never ported to Python 3. ([\#4334](https://github.com/matrix-org/synapse/issues/4334))

Internal Changes
----------------

- Getting URL previews of IP addresses no longer fails on Python 3. ([\#4215](https://github.com/matrix-org/synapse/issues/4215))
- drop undocumented dependency on dateutil ([\#4266](https://github.com/matrix-org/synapse/issues/4266))
- Update the example systemd config to use a virtualenv ([\#4273](https://github.com/matrix-org/synapse/issues/4273))
- Update link to kernel DCO guide ([\#4274](https://github.com/matrix-org/synapse/issues/4274))
- Make isort tox check print diff when it fails ([\#4283](https://github.com/matrix-org/synapse/issues/4283))
- Log room_id in Unknown room errors ([\#4297](https://github.com/matrix-org/synapse/issues/4297))
- Documentation improvements for coturn setup. Contributed by Krithin Sitaram. ([\#4333](https://github.com/matrix-org/synapse/issues/4333))
- Update pull request template to use absolute links ([\#4341](https://github.com/matrix-org/synapse/issues/4341))
- Update README to not lie about required restart when updating TLS certificates ([\#4343](https://github.com/matrix-org/synapse/issues/4343))
- Update debian packaging for compatibility with transitional package ([\#4349](https://github.com/matrix-org/synapse/issues/4349))
- Fix command hint to generate a config file when trying to start without a config file ([\#4353](https://github.com/matrix-org/synapse/issues/4353))
- Add better logging for unexpected errors while sending transactions ([\#4358](https://github.com/matrix-org/synapse/issues/4358))

1 files changed, 108 insertions, 143 deletions

diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py

index 92422c6ffc..69c5f9fe2e 100644
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -15,173 +15,138 @@
 # limitations under the License.
 
 import logging
-from distutils.version import LooseVersion
+
+from pkg_resources import DistributionNotFound, VersionConflict, get_distribution
 
 logger = logging.getLogger(__name__)
 
-# this dict maps from python package name to a list of modules we expect it to
-# provide.
-#
-# the key is a "requirement specifier", as used as a parameter to `pip
-# install`[1], or an `install_requires` argument to `setuptools.setup` [2].
+
+# REQUIREMENTS is a simple list of requirement specifiers[1], and must be
+# installed. It is passed to setup() as install_requires in setup.py.
 #
-# the value is a sequence of strings; each entry should be the name of the
-# python module, optionally followed by a version assertion which can be either
-# ">=<ver>" or "==<ver>".
+# CONDITIONAL_REQUIREMENTS is the optional dependencies, represented as a dict
+# of lists. The dict key is the optional dependency name and can be passed to
+# pip when installing. The list is a series of requirement specifiers[1] to be
+# installed when that optional dependency requirement is specified. It is passed
+# to setup() as extras_require in setup.py
 #
 # [1] https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers.
-# [2] https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-dependencies
-REQUIREMENTS = {
-    "jsonschema>=2.5.1": ["jsonschema>=2.5.1"],
-    "frozendict>=1": ["frozendict"],
-    "unpaddedbase64>=1.1.0": ["unpaddedbase64>=1.1.0"],
-    "canonicaljson>=1.1.3": ["canonicaljson>=1.1.3"],
-    "signedjson>=1.0.0": ["signedjson>=1.0.0"],
-    "pynacl>=1.2.1": ["nacl>=1.2.1", "nacl.bindings"],
-    "service_identity>=16.0.0": ["service_identity>=16.0.0"],
-    "Twisted>=17.1.0": ["twisted>=17.1.0"],
-    "treq>=15.1": ["treq>=15.1"],
 
+REQUIREMENTS = [
+    "jsonschema>=2.5.1",
+    "frozendict>=1",
+    "unpaddedbase64>=1.1.0",
+    "canonicaljson>=1.1.3",
+    "signedjson>=1.0.0",
+    "pynacl>=1.2.1",
+    "service_identity>=16.0.0",
+    "Twisted>=17.1.0",
+    "treq>=15.1",
     # Twisted has required pyopenssl 16.0 since about Twisted 16.6.
-    "pyopenssl>=16.0.0": ["OpenSSL>=16.0.0"],
-
-    "pyyaml>=3.11": ["yaml"],
-    "pyasn1>=0.1.9": ["pyasn1"],
-    "pyasn1-modules>=0.0.7": ["pyasn1_modules"],
-    "daemonize>=2.3.1": ["daemonize"],
-    "bcrypt>=3.1.0": ["bcrypt>=3.1.0"],
-    "pillow>=3.1.2": ["PIL"],
-    "sortedcontainers>=1.4.4": ["sortedcontainers"],
-    "psutil>=2.0.0": ["psutil>=2.0.0"],
-    "pysaml2>=3.0.0": ["saml2"],
-    "pymacaroons-pynacl>=0.9.3": ["pymacaroons"],
-    "msgpack-python>=0.4.2": ["msgpack"],
-    "phonenumbers>=8.2.0": ["phonenumbers"],
-    "six>=1.10": ["six"],
-
+    "pyopenssl>=16.0.0",
+    "pyyaml>=3.11",
+    "pyasn1>=0.1.9",
+    "pyasn1-modules>=0.0.7",
+    "daemonize>=2.3.1",
+    "bcrypt>=3.1.0",
+    "pillow>=3.1.2",
+    "sortedcontainers>=1.4.4",
+    "psutil>=2.0.0",
+    "pymacaroons-pynacl>=0.9.3",
+    "msgpack-python>=0.4.2",
+    "phonenumbers>=8.2.0",
+    "six>=1.10",
     # prometheus_client 0.4.0 changed the format of counter metrics
     # (cf https://github.com/matrix-org/synapse/issues/4001)
-    "prometheus_client>=0.0.18,<0.4.0": ["prometheus_client"],
-
+    "prometheus_client>=0.0.18,<0.4.0",
     # we use attr.s(slots), which arrived in 16.0.0
-    "attrs>=16.0.0": ["attr>=16.0.0"],
-    "netaddr>=0.7.18": ["netaddr"],
-}
+    "attrs>=16.0.0",
+    "netaddr>=0.7.18",
+]
 
 CONDITIONAL_REQUIREMENTS = {
-    "email.enable_notifs": {
-        "Jinja2>=2.8": ["Jinja2>=2.8"],
-        "bleach>=1.4.2": ["bleach>=1.4.2"],
-    },
-    "matrix-synapse-ldap3": {
-        "matrix-synapse-ldap3>=0.1": ["ldap_auth_provider"],
-    },
-    "postgres": {
-        "psycopg2>=2.6": ["psycopg2"]
-    }
-}
-
-
-def requirements(config=None, include_conditional=False):
-    reqs = REQUIREMENTS.copy()
-    if include_conditional:
-        for _, req in CONDITIONAL_REQUIREMENTS.items():
-            reqs.update(req)
-    return reqs
+    "email.enable_notifs": ["Jinja2>=2.9", "bleach>=1.4.2"],
+    "matrix-synapse-ldap3": ["matrix-synapse-ldap3>=0.1"],
+    "postgres": ["psycopg2>=2.6"],
 
+    # ConsentResource uses select_autoescape, which arrived in jinja 2.9
+    "resources.consent": ["Jinja2>=2.9"],
 
-def github_link(project, version, egg):
-    return "https://github.com/%s/tarball/%s/#egg=%s" % (project, version, egg)
-
-
-DEPENDENCY_LINKS = {
+    "saml2": ["pysaml2>=4.5.0"],
+    "url_preview": ["lxml>=3.5.0"],
+    "test": ["mock>=2.0"],
 }
 
 
-class MissingRequirementError(Exception):
-    def __init__(self, message, module_name, dependency):
-        super(MissingRequirementError, self).__init__(message)
-        self.module_name = module_name
-        self.dependency = dependency
-
-
-def check_requirements(config=None):
-    """Checks that all the modules needed by synapse have been correctly
-    installed and are at the correct version"""
-    for dependency, module_requirements in (
-            requirements(config, include_conditional=False).items()):
-        for module_requirement in module_requirements:
-            if ">=" in module_requirement:
-                module_name, required_version = module_requirement.split(">=")
-                version_test = ">="
-            elif "==" in module_requirement:
-                module_name, required_version = module_requirement.split("==")
-                version_test = "=="
-            else:
-                module_name = module_requirement
-                version_test = None
-
-            try:
-                module = __import__(module_name)
-            except ImportError:
-                logging.exception(
-                    "Can't import %r which is part of %r",
-                    module_name, dependency
-                )
-                raise MissingRequirementError(
-                    "Can't import %r which is part of %r"
-                    % (module_name, dependency), module_name, dependency
-                )
-            version = getattr(module, "__version__", None)
-            file_path = getattr(module, "__file__", None)
-            logger.info(
-                "Using %r version %r from %r to satisfy %r",
-                module_name, version, file_path, dependency
+def list_requirements():
+    deps = set(REQUIREMENTS)
+    for opt in CONDITIONAL_REQUIREMENTS.values():
+        deps = set(opt) | deps
+
+    return list(deps)
+
+
+class DependencyException(Exception):
+    @property
+    def message(self):
+        return "\n".join([
+            "Missing Requirements: %s" % (", ".join(self.dependencies),),
+            "To install run:",
+            "    pip install --upgrade --force %s" % (" ".join(self.dependencies),),
+            "",
+        ])
+
+    @property
+    def dependencies(self):
+        for i in self.args[0]:
+            yield '"' + i + '"'
+
+
+def check_requirements(for_feature=None, _get_distribution=get_distribution):
+    deps_needed = []
+    errors = []
+
+    if for_feature:
+        reqs = CONDITIONAL_REQUIREMENTS[for_feature]
+    else:
+        reqs = REQUIREMENTS
+
+    for dependency in reqs:
+        try:
+            _get_distribution(dependency)
+        except VersionConflict as e:
+            deps_needed.append(dependency)
+            errors.append(
+                "Needed %s, got %s==%s"
+                % (dependency, e.dist.project_name, e.dist.version)
             )
+        except DistributionNotFound:
+            deps_needed.append(dependency)
+            errors.append("Needed %s but it was not installed" % (dependency,))
 
-            if version_test == ">=":
-                if version is None:
-                    raise MissingRequirementError(
-                        "Version of %r isn't set as __version__ of module %r"
-                        % (dependency, module_name), module_name, dependency
-                    )
-                if LooseVersion(version) < LooseVersion(required_version):
-                    raise MissingRequirementError(
-                        "Version of %r in %r is too old. %r < %r"
-                        % (dependency, file_path, version, required_version),
-                        module_name, dependency
-                    )
-            elif version_test == "==":
-                if version is None:
-                    raise MissingRequirementError(
-                        "Version of %r isn't set as __version__ of module %r"
-                        % (dependency, module_name), module_name, dependency
-                    )
-                if LooseVersion(version) != LooseVersion(required_version):
-                    raise MissingRequirementError(
-                        "Unexpected version of %r in %r. %r != %r"
-                        % (dependency, file_path, version, required_version),
-                        module_name, dependency
-                    )
+    if not for_feature:
+        # Check the optional dependencies are up to date. We allow them to not be
+        # installed.
+        OPTS = sum(CONDITIONAL_REQUIREMENTS.values(), [])
 
+        for dependency in OPTS:
+            try:
+                _get_distribution(dependency)
+            except VersionConflict:
+                deps_needed.append(dependency)
+                errors.append("Needed %s but it was not installed" % (dependency,))
+            except DistributionNotFound:
+                # If it's not found, we don't care
+                pass
 
-def list_requirements():
-    result = []
-    linked = []
-    for link in DEPENDENCY_LINKS.values():
-        egg = link.split("#egg=")[1]
-        linked.append(egg.split('-')[0])
-        result.append(link)
-    for requirement in requirements(include_conditional=True):
-        is_linked = False
-        for link in linked:
-            if requirement.replace('-', '_').startswith(link):
-                is_linked = True
-        if not is_linked:
-            result.append(requirement)
-    return result
+    if deps_needed:
+        for e in errors:
+            logging.error(e)
+
+        raise DependencyException(deps_needed)
 
 
 if __name__ == "__main__":
     import sys
+
     sys.stdout.writelines(req + "\n" for req in list_requirements())