Merge branch 'release-v1.38' into develop

36 files changed, 87 insertions, 64 deletions

diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml
index c239130c57..808f825331 100644
--- a/.github/workflows/docs.yaml
+++ b/.github/workflows/docs.yaml
@@ -7,6 +7,8 @@ on:
       - develop
       # For documentation specific to a release
       - 'release-v*'
+      # stable docs
+      - master
 
   workflow_dispatch:
 
@@ -30,40 +32,35 @@ jobs:
           mdbook build
           cp book/welcome_and_overview.html book/index.html
 
-      # Deploy to the latest documentation directories
-      - name: Deploy latest documentation
-        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          keep_files: true
-          publish_dir: ./book
-          destination_dir: ./develop
-
-      - name: Get the current Synapse version
+      # Figure out the target directory.
+      #
+      # The target directory depends on the name of the branch
+      #
+      - name: Get the target directory name
         id: vars
-        # The $GITHUB_REF value for a branch looks like `refs/heads/release-v1.2`. We do some
-        # shell magic to remove the "refs/heads/release-v" bit from this, to end up with "1.2",
-        # our major/minor version number, and set this to a var called `branch-version`.
-        #
-        # We then use some python to get Synapse's full version string, which may look
-        # like "1.2.3rc4". We set this to a var called `synapse-version`. We use this
-        # to determine if this release is still an RC, and if so block deployment.
         run: |
-          echo ::set-output name=branch-version::${GITHUB_REF#refs/heads/release-v}
-          echo ::set-output name=synapse-version::`python3 -c 'import synapse; print(synapse.__version__)'`
+          # first strip the 'refs/heads/' prefix with some shell foo
+          branch="${GITHUB_REF#refs/heads/}"
 
-      # Deploy to the version-specific directory
-      - name: Deploy release-specific documentation
-        # We only carry out this step if we're running on a release branch,
-        # and the current Synapse version does not have "rc" in the name.
-        #
-        # The result is that only full releases are deployed, but can be
-        # updated if the release branch gets retroactive fixes.
-        if: ${{ startsWith( github.ref, 'refs/heads/release-v' ) && !contains( steps.vars.outputs.synapse-version, 'rc') }}
-        uses: peaceiris/actions-gh-pages@v3
+          case $branch in
+              release-*)
+                  # strip 'release-' from the name for release branches.
+                  branch="${branch#release-}"
+                  ;;
+              master)
+                  # deploy to "latest" for the master branch.
+                  branch="latest"
+                  ;;
+          esac
+
+          # finally, set the 'branch-version' var.
+          echo "::set-output name=branch-version::$branch"
+          
+      # Deploy to the target directory.
+      - name: Deploy to gh pages
+        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           keep_files: true
           publish_dir: ./book
-          # The resulting documentation will end up in a directory named `vX.Y`.
-          destination_dir: ./v${{ steps.vars.outputs.branch-version }}
+          destination_dir: ./${{ steps.vars.outputs.branch-version }}
diff --git a/CHANGES.md b/CHANGES.md
index a2fc423096..c4551fdd69 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,7 +1,54 @@
-Synapse 1.38.0 (**UNRELEASED**)
-===============================
+Synapse 1.38.0rc1 (2021-07-06)
+==============================
+
 This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade.md#upgrading-to-v1380) for more information.
 
+Features
+--------
+
+- Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450))
+- Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205))
+- Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214))
+- Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243))
+- Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261))
+- Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263))
+- Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284))
+- Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223))
+- Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252))
+- Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314))
+- Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279))
+- Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303))
+
+
+Improved Documentation
+----------------------
+
+- Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166))
+- Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242))
+- Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258))
+- Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288))
+- Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302))
+
+
+Internal Changes
+----------------
+
+- Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114))
+- Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213))
+- Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237))
+- Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239))
+- Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253))
+- Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256))
+- Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268))
+- Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))
+
+
 Synapse 1.37.1 (2021-06-30)
 ===========================
 
diff --git a/changelog.d/10114.misc b/changelog.d/10114.misc
deleted file mode 100644
index 808548f7c7..0000000000
--- a/changelog.d/10114.misc
+++ /dev/null
@@ -1 +0,0 @@
-Drop Origin and Accept from the value of the Access-Control-Allow-Headers response header.
diff --git a/changelog.d/10166.doc b/changelog.d/10166.doc
deleted file mode 100644
index 8d1710c132..0000000000
--- a/changelog.d/10166.doc
+++ /dev/null
@@ -1 +0,0 @@
-Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown.
diff --git a/changelog.d/10205.feature b/changelog.d/10205.feature
deleted file mode 100644
index db3fd22587..0000000000
--- a/changelog.d/10205.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for evicting cache entries based on last access time.
diff --git a/changelog.d/10213.misc b/changelog.d/10213.misc
deleted file mode 100644
index 9adb0fbd02..0000000000
--- a/changelog.d/10213.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to the federation servlets.
diff --git a/changelog.d/10214.feature b/changelog.d/10214.feature
deleted file mode 100644
index a3818c9d25..0000000000
--- a/changelog.d/10214.feature
+++ /dev/null
@@ -1 +0,0 @@
-Omit empty fields from the `/sync` response. Contributed by @deepbluev7.
\ No newline at end of file
diff --git a/changelog.d/10223.bugfix b/changelog.d/10223.bugfix
deleted file mode 100644
index 4e42f6b608..0000000000
--- a/changelog.d/10223.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner.
diff --git a/changelog.d/10225.feature b/changelog.d/10225.feature
deleted file mode 100644
index d16f66ffe9..0000000000
--- a/changelog.d/10225.feature
+++ /dev/null
@@ -1 +0,0 @@
-Improve validation on federation `send_{join,leave,knock}` endpoints.
diff --git a/changelog.d/10237.misc b/changelog.d/10237.misc
deleted file mode 100644
index d76c119a41..0000000000
--- a/changelog.d/10237.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the reliability of auto-joining remote rooms.
diff --git a/changelog.d/10239.misc b/changelog.d/10239.misc
deleted file mode 100644
index d05f1c4411..0000000000
--- a/changelog.d/10239.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update the release script to use the semver terminology and determine the release branch based on the next version.
diff --git a/changelog.d/10242.doc b/changelog.d/10242.doc
deleted file mode 100644
index 2241b28547..0000000000
--- a/changelog.d/10242.doc
+++ /dev/null
@@ -1 +0,0 @@
-Choose Welcome & Overview as the default page for synapse documentation website.
diff --git a/changelog.d/10243.feature b/changelog.d/10243.feature
deleted file mode 100644
index d16f66ffe9..0000000000
--- a/changelog.d/10243.feature
+++ /dev/null
@@ -1 +0,0 @@
-Improve validation on federation `send_{join,leave,knock}` endpoints.
diff --git a/changelog.d/10252.bugfix b/changelog.d/10252.bugfix
deleted file mode 100644
index c8ddd14528..0000000000
--- a/changelog.d/10252.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled.
diff --git a/changelog.d/10253.misc b/changelog.d/10253.misc
deleted file mode 100644
index 44d9217245..0000000000
--- a/changelog.d/10253.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix type hints for computing auth events.
diff --git a/changelog.d/10256.misc b/changelog.d/10256.misc
deleted file mode 100644
index adef12fcb9..0000000000
--- a/changelog.d/10256.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general).
diff --git a/changelog.d/10258.doc b/changelog.d/10258.doc
deleted file mode 100644
index 1549786c0c..0000000000
--- a/changelog.d/10258.doc
+++ /dev/null
@@ -1 +0,0 @@
-Adjust the URL in the README.rst file to point to irc.libera.chat.
diff --git a/changelog.d/10261.feature b/changelog.d/10261.feature
deleted file mode 100644
index cd55cecbd5..0000000000
--- a/changelog.d/10261.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add SSO `external_ids` to the Query User Account admin API.
diff --git a/changelog.d/10263.feature b/changelog.d/10263.feature
deleted file mode 100644
index 7b1d2fe60f..0000000000
--- a/changelog.d/10263.feature
+++ /dev/null
@@ -1 +0,0 @@
-Mark events received over federation which fail a spam check as "soft-failed".
diff --git a/changelog.d/10267.bugfix b/changelog.d/10267.bugfix
deleted file mode 100644
index 7ebda7cdc2..0000000000
--- a/changelog.d/10267.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server.
diff --git a/changelog.d/10268.misc b/changelog.d/10268.misc
deleted file mode 100644
index 9e3f60c72f..0000000000
--- a/changelog.d/10268.misc
+++ /dev/null
@@ -1 +0,0 @@
-Move event authentication methods from `Auth` to `EventAuthHandler`.
diff --git a/changelog.d/10279.bugfix b/changelog.d/10279.bugfix
deleted file mode 100644
index ac8b64ead9..0000000000
--- a/changelog.d/10279.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1.
diff --git a/changelog.d/10282.bugfix b/changelog.d/10282.bugfix
deleted file mode 100644
index 7ebda7cdc2..0000000000
--- a/changelog.d/10282.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server.
diff --git a/changelog.d/10284.feature b/changelog.d/10284.feature
deleted file mode 100644
index 379155e8cf..0000000000
--- a/changelog.d/10284.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add metrics for new inbound federation staging area.
diff --git a/changelog.d/10286.bugfix b/changelog.d/10286.bugfix
deleted file mode 100644
index 7ebda7cdc2..0000000000
--- a/changelog.d/10286.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server.
diff --git a/changelog.d/10288.doc b/changelog.d/10288.doc
deleted file mode 100644
index 0739687b92..0000000000
--- a/changelog.d/10288.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix homeserver config option name in presence router documentation.
diff --git a/changelog.d/10290.feature b/changelog.d/10290.feature
deleted file mode 100644
index 4e4c2e24ef..0000000000
--- a/changelog.d/10290.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add script to print information about recently registered users.
diff --git a/changelog.d/10291.bugfix b/changelog.d/10291.bugfix
deleted file mode 100644
index 7ebda7cdc2..0000000000
--- a/changelog.d/10291.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server.
diff --git a/changelog.d/10292.misc b/changelog.d/10292.misc
deleted file mode 100644
index 9e87d8682c..0000000000
--- a/changelog.d/10292.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reenable a SyTest after it has been fixed.
diff --git a/changelog.d/10302.doc b/changelog.d/10302.doc
deleted file mode 100644
index 7386817de7..0000000000
--- a/changelog.d/10302.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix link pointing at the wrong section in the modules documentation page.
diff --git a/changelog.d/10303.bugfix b/changelog.d/10303.bugfix
deleted file mode 100644
index c0577c9f73..0000000000
--- a/changelog.d/10303.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up.
diff --git a/changelog.d/10314.bugfix b/changelog.d/10314.bugfix
deleted file mode 100644
index 7ebda7cdc2..0000000000
--- a/changelog.d/10314.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server.
diff --git a/changelog.d/10264.bugfix b/changelog.d/10326.bugfix
index 7ebda7cdc2..7ebda7cdc2 100644
--- a/changelog.d/10264.bugfix
+++ b/changelog.d/10326.bugfix
diff --git a/changelog.d/9450.feature b/changelog.d/9450.feature
deleted file mode 100644
index 455936a41d..0000000000
--- a/changelog.d/9450.feature
+++ /dev/null
@@ -1 +0,0 @@
-Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918).
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 1bd03462ac..aa9a3269c0 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.37.1"
+__version__ = "1.38.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/storage/databases/main/events_bg_updates.py b/synapse/storage/databases/main/events_bg_updates.py
index 1c95c66648..29f33bac55 100644
--- a/synapse/storage/databases/main/events_bg_updates.py
+++ b/synapse/storage/databases/main/events_bg_updates.py
@@ -1146,6 +1146,16 @@ class EventsBackgroundUpdatesStore(SQLBaseStore):
                 logger.info("completing stream_ordering migration: %s", sql)
                 txn.execute(sql)
 
+        # ANALYZE the new column to build stats on it, to encourage PostgreSQL to use the
+        # indexes on it.
+        # We need to pass execute a dummy function to handle the txn's result otherwise
+        # it tries to call fetchall() on it and fails because there's no result to fetch.
+        await self.db_pool.execute(
+            "background_analyze_new_stream_ordering_column",
+            lambda txn: None,
+            "ANALYZE events(stream_ordering2)",
+        )
+
         await self.db_pool.runInteraction(
             "_background_replace_stream_ordering_column", process
         )