Merge tag 'v1.38.0' into babolivier/dinsic_1.41.0

Synapse 1.38.0 (2021-07-13)
===========================

This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information.

No significant changes since 1.38.0rc3.

Synapse 1.38.0rc3 (2021-07-13)
==============================

Internal Changes
----------------

- Build the Debian packages in CI. ([\#10247](https://github.com/matrix-org/synapse/issues/10247), [\#10379](https://github.com/matrix-org/synapse/issues/10379))

Synapse 1.38.0rc2 (2021-07-09)
==============================

Bugfixes
--------

- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))

Improved Documentation
----------------------

- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))

Synapse 1.38.0rc1 (2021-07-06)
==============================

Features
--------

- Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450))
- Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205))
- Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214))
- Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243))
- Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261))
- Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263))
- Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284))
- Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290))

Bugfixes
--------

- Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223))
- Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252))
- Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326))
- Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279))
- Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303))

Improved Documentation
----------------------

- Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166))
- Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242))
- Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258))
- Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288))
- Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302))

Internal Changes
----------------

- Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114))
- Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213))
- Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237))
- Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239))
- Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253))
- Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256))
- Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268))
- Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))

3 files changed, 41 insertions, 30 deletions

diff --git a/scripts-dev/build_debian_packages b/scripts-dev/build_debian_packages

index 546724f89f..e25c5bb265 100755
--- a/scripts-dev/build_debian_packages
+++ b/scripts-dev/build_debian_packages
@@ -10,6 +10,7 @@
 # can be passed on the commandline for debugging.
 
 import argparse
+import json
 import os
 import signal
 import subprocess
@@ -34,6 +35,8 @@ By default, builds for all known distributions, but a list of distributions
 can be passed on the commandline for debugging.
 """
 
+projdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+
 
 class Builder(object):
     def __init__(self, redirect_stdout=False):
@@ -57,9 +60,6 @@ class Builder(object):
             raise
 
     def _inner_build(self, dist, skip_tests=False):
-        projdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-        os.chdir(projdir)
-
         tag = dist.split(":", 1)[1]
 
         # Make the dir where the debs will live.
@@ -93,6 +93,7 @@ class Builder(object):
             ],
             stdout=stdout,
             stderr=subprocess.STDOUT,
+            cwd=projdir,
         )
 
         container_name = "synapse_build_" + tag
@@ -180,10 +181,18 @@ if __name__ == "__main__":
         help="skip running tests after building",
     )
     parser.add_argument(
+        "--show-dists-json",
+        action="store_true",
+        help="instead of building the packages, just list the dists to build for, as a json array",
+    )
+    parser.add_argument(
         "dist",
         nargs="*",
         default=DISTS,
         help="a list of distributions to build for. Default: %(default)s",
     )
     args = parser.parse_args()
-    run_builds(dists=args.dist, jobs=args.jobs, skip_tests=args.no_check)
+    if args.show_dists_json:
+        print(json.dumps(DISTS))
+    else:
+        run_builds(dists=args.dist, jobs=args.jobs, skip_tests=args.no_check)
diff --git a/scripts-dev/complement.sh b/scripts-dev/complement.sh

index ba060104c3..aca32edc17 100755
--- a/scripts-dev/complement.sh
+++ b/scripts-dev/complement.sh
@@ -65,4 +65,4 @@ if [[ -n "$1" ]]; then
 fi
 
 # Run the tests!
-go test -v -tags synapse_blacklist,msc2946,msc3083,msc2716 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests
+go test -v -tags synapse_blacklist,msc2946,msc3083,msc2716,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests
diff --git a/scripts-dev/release.py b/scripts-dev/release.py

index fc3df9071c..5bfaa4ad2f 100755
--- a/scripts-dev/release.py
+++ b/scripts-dev/release.py
@@ -83,12 +83,6 @@ def run():
     if current_version.pre:
         # If the current version is an RC we don't need to bump any of the
         # version numbers (other than the RC number).
-        base_version = "{}.{}.{}".format(
-            current_version.major,
-            current_version.minor,
-            current_version.micro,
-        )
-
         if rc:
             new_version = "{}.{}.{}rc{}".format(
                 current_version.major,
@@ -97,49 +91,57 @@ def run():
                 current_version.pre[1] + 1,
             )
         else:
-            new_version = base_version
+            new_version = "{}.{}.{}".format(
+                current_version.major,
+                current_version.minor,
+                current_version.micro,
+            )
     else:
-        # If this is a new release cycle then we need to know if its a major
-        # version bump or a hotfix.
+        # If this is a new release cycle then we need to know if it's a minor
+        # or a patch version bump.
         release_type = click.prompt(
             "Release type",
-            type=click.Choice(("major", "hotfix")),
+            type=click.Choice(("minor", "patch")),
             show_choices=True,
-            default="major",
+            default="minor",
         )
 
-        if release_type == "major":
-            base_version = new_version = "{}.{}.{}".format(
-                current_version.major,
-                current_version.minor + 1,
-                0,
-            )
+        if release_type == "minor":
             if rc:
                 new_version = "{}.{}.{}rc1".format(
                     current_version.major,
                     current_version.minor + 1,
                     0,
                 )
-
+            else:
+                new_version = "{}.{}.{}".format(
+                    current_version.major,
+                    current_version.minor + 1,
+                    0,
+                )
         else:
-            base_version = new_version = "{}.{}.{}".format(
-                current_version.major,
-                current_version.minor,
-                current_version.micro + 1,
-            )
             if rc:
                 new_version = "{}.{}.{}rc1".format(
                     current_version.major,
                     current_version.minor,
                     current_version.micro + 1,
                 )
+            else:
+                new_version = "{}.{}.{}".format(
+                    current_version.major,
+                    current_version.minor,
+                    current_version.micro + 1,
+                )
 
     # Confirm the calculated version is OK.
     if not click.confirm(f"Create new version: {new_version}?", default=True):
         click.get_current_context().abort()
 
     # Switch to the release branch.
-    release_branch_name = f"release-v{current_version.major}.{current_version.minor}"
+    parsed_new_version = version.parse(new_version)
+    release_branch_name = (
+        f"release-v{parsed_new_version.major}.{parsed_new_version.minor}"
+    )
     release_branch = find_ref(repo, release_branch_name)
     if release_branch:
         if release_branch.is_remote():
@@ -153,7 +155,7 @@ def run():
         # release type.
         if current_version.is_prerelease:
             default = release_branch_name
-        elif release_type == "major":
+        elif release_type == "minor":
             default = "develop"
         else:
             default = "master"