From 6c994913473b70e164b13a4f551da8a8d448cc33 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Thu, 23 Apr 2015 16:07:49 +0100 Subject: prometheus/metrics howto from Leo --- docs/metrics-howto.rst | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/metrics-howto.rst (limited to 'docs') diff --git a/docs/metrics-howto.rst b/docs/metrics-howto.rst new file mode 100644 index 0000000000..b3e71fc770 --- /dev/null +++ b/docs/metrics-howto.rst @@ -0,0 +1,49 @@ +How to monitor Synapse metrics using Prometheus +=============================================== + +1: install prometheus: + Follow instructions at http://prometheus.io/docs/introduction/install/ + +2: enable synapse metrics: + Simply setting a (local) port number will enable it. Pick a port. + prometheus itself defaults to 9090, so starting just above that for + locally monitored services seems reasonable. E.g. 9092: + + Add to homeserver.yaml + + metrics_port: 9092 + + Restart synapse + +3: check out synapse-prometheus-config + https://github.com/matrix-org/synapse-prometheus-config + +4: arrange for synapse.html to appear in prometheus's "consoles" + directory - symlink might be easiest to ensure `git pull` keeps it + updated. + +5: arrange for synapse.rules to be invoked from the main + prometheus.conf and add a synapse target. This is easiest if + prometheus runs on the same machine as synapse, as it can then just + use localhost:: + + global: { + rule_file: "synapse.rules" + } + + job: { + name: "synapse" + + target_group: { + target: "http://localhost:9092/" + } + } + +6: start prometheus:: + + ./prometheus -config.file=prometheus.conf + +7: wait a few seconds for it to start and perform the first scrape, + then visit the console: + + http://server-where-prometheus-runs:9090/consoles/synapse.html -- cgit 1.4.1 From 8c784142845bf462b255374b4cbacc22fd572847 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Thu, 23 Apr 2015 16:14:08 +0100 Subject: Formatting / wording fixes to metrics doc --- docs/metrics-howto.rst | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) (limited to 'docs') diff --git a/docs/metrics-howto.rst b/docs/metrics-howto.rst index b3e71fc770..99776cd504 100644 --- a/docs/metrics-howto.rst +++ b/docs/metrics-howto.rst @@ -1,10 +1,10 @@ How to monitor Synapse metrics using Prometheus =============================================== -1: install prometheus: +1: Install prometheus: Follow instructions at http://prometheus.io/docs/introduction/install/ -2: enable synapse metrics: +2: Enable synapse metrics: Simply setting a (local) port number will enable it. Pick a port. prometheus itself defaults to 9090, so starting just above that for locally monitored services seems reasonable. E.g. 9092: @@ -15,17 +15,18 @@ How to monitor Synapse metrics using Prometheus Restart synapse -3: check out synapse-prometheus-config +3: Check out synapse-prometheus-config https://github.com/matrix-org/synapse-prometheus-config -4: arrange for synapse.html to appear in prometheus's "consoles" - directory - symlink might be easiest to ensure `git pull` keeps it - updated. +4: Add `synapse.html` and `synapse.rules` + The `.html` file needs to appear in prometheus's "consoles" directory, and + the `.rules` file needs to be invoked somewhere in the main config file. + A symlink to each from the git checkout into the prometheus directory might be + easiest to ensure `git pull` keeps it updated. -5: arrange for synapse.rules to be invoked from the main - prometheus.conf and add a synapse target. This is easiest if - prometheus runs on the same machine as synapse, as it can then just - use localhost:: +5: Add a prometheus target for synapse + This is easiest if prometheus runs on the same machine as synapse, as it can + then just use localhost:: global: { rule_file: "synapse.rules" @@ -39,11 +40,11 @@ How to monitor Synapse metrics using Prometheus } } -6: start prometheus:: +6: Start prometheus:: ./prometheus -config.file=prometheus.conf -7: wait a few seconds for it to start and perform the first scrape, +7: Wait a few seconds for it to start and perform the first scrape, then visit the console: http://server-where-prometheus-runs:9090/consoles/synapse.html -- cgit 1.4.1 From 6d1540134133cfe07fbecfbf0c733aceade33a05 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Thu, 23 Apr 2015 16:16:08 +0100 Subject: Mumble ReST mumble ``fixed-width`` mumble --- docs/metrics-howto.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'docs') diff --git a/docs/metrics-howto.rst b/docs/metrics-howto.rst index 99776cd504..c1f5ae2174 100644 --- a/docs/metrics-howto.rst +++ b/docs/metrics-howto.rst @@ -18,11 +18,11 @@ How to monitor Synapse metrics using Prometheus 3: Check out synapse-prometheus-config https://github.com/matrix-org/synapse-prometheus-config -4: Add `synapse.html` and `synapse.rules` - The `.html` file needs to appear in prometheus's "consoles" directory, and - the `.rules` file needs to be invoked somewhere in the main config file. - A symlink to each from the git checkout into the prometheus directory might be - easiest to ensure `git pull` keeps it updated. +4: Add ``synapse.html`` and ``synapse.rules`` + The ``.html`` file needs to appear in prometheus's ``consoles`` directory, + and the ``.rules`` file needs to be invoked somewhere in the main config + file. A symlink to each from the git checkout into the prometheus directory + might be easiest to ensure ``git pull`` keeps it updated. 5: Add a prometheus target for synapse This is easiest if prometheus runs on the same machine as synapse, as it can -- cgit 1.4.1 From 56f518d279b642efce92e172b463232937d50b8c Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 27 Apr 2015 14:53:35 +0100 Subject: Add docs on how to use synapse with psycopg2 --- docs/postgres.rst | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 docs/postgres.rst (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst new file mode 100644 index 0000000000..5bb45d3ed9 --- /dev/null +++ b/docs/postgres.rst @@ -0,0 +1,34 @@ +Using Postgres +-------------- + +Set up client +============= +We need to have installed the postgres python connector ``psycopg2``. In the +virtual env:: + + sudo apt-get install libpq-dev + pip install psycopg2 + + +Synapse config +============== + +Add the following line to your config file:: + + database_config: + +Where ```` is the file name that points to a yaml file of the +following form:: + + name: psycopg2 + args: + user: + password: + database: + host: + cp_min: 5 + cp_max: 10 + +All key, values in ``args`` are passed to the ``psycopg2.connect(..)`` +function, except keys beginning with ``cp_``, which are consumed by the twisted +adbapi connection pool. -- cgit 1.4.1 From 3151afee9e3213ba439f701e9b968a32ec41054d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Apr 2015 17:59:27 +0100 Subject: Update docs/postgres.rst to explain port script usage --- docs/postgres.rst | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst index 5bb45d3ed9..bbf1b12a89 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -32,3 +32,54 @@ following form:: All key, values in ``args`` are passed to the ``psycopg2.connect(..)`` function, except keys beginning with ``cp_``, which are consumed by the twisted adbapi connection pool. + + +Porting from SQLite +=================== + +The script ``port_from_sqlite_to_postgres.py`` allows porting an existing +synapse server backed by SQLite to using PostgreSQL. This is done in as a two +phase process: + +1. Copy the existing SQLite database to a separate location (while the server + is down) and running the port script against that offline database. +2. Shut down the server. Rerun the port script to port any data that has come + in since taking the first snapshot. Restart server against the PostgrSQL + database. + +The port script is designed to be run repeatedly against newer snapshots of the +SQLite database file. This makes it safe to repeat step 1 if there was a delay +between taking the previous snapshot and ready to do step 2. + +It is safe to at any time kill the port script and restart it. + +Using the port script +~~~~~~~~~~~~~~~~~~~~~ + +Firstly, shut down the currently running synapse server and copy its database +file to another location. Once the copy is complete, restart synapse. + +Assuming your database config file (as described in the section *Synapse +config*) is named ``database_config.yaml`` and the SQLite snapshot is at +``homeserver.db.snapshot`` then simply run:: + + python scripts/port_from_sqlite_to_postgres.py \ + --sqlite-database homeserver.db.snapshot \ + --postgres-config database_config.yaml + +The flag ``--curses`` displays a coloured curses progress UI. + +If the script took a long time to complete, or time has otherwise passed since +the original snapshot was taken, repeat the previous steps with a newer +snapshot. + +To complete the conversion shut down the synapse server and run the port +script one last time, e.g. if the SQLite database is at ``homeserver.db`` run: + + python scripts/port_from_sqlite_to_postgres.py \ + --sqlite-database homeserver.db \ + --postgres-config database_config.yaml + +Once that has completed, change the synapse config to point at the PostgreSQL +database configuration file and restart synapse. Synapse should now be running +against PostgreSQL. -- cgit 1.4.1 From cc52f02d74e5eee4eb64ff16fb0dfaa7d4d7d38e Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Apr 2015 18:09:20 +0100 Subject: Fix rst --- docs/postgres.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst index bbf1b12a89..ca72131c6c 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -74,7 +74,8 @@ the original snapshot was taken, repeat the previous steps with a newer snapshot. To complete the conversion shut down the synapse server and run the port -script one last time, e.g. if the SQLite database is at ``homeserver.db`` run: +script one last time, e.g. if the SQLite database is at ``homeserver.db`` +run:: python scripts/port_from_sqlite_to_postgres.py \ --sqlite-database homeserver.db \ -- cgit 1.4.1 From 478e511db04548023f5a652e6ed7117ecd3d3cf5 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Wed, 29 Apr 2015 00:48:07 +0100 Subject: improve postgres blurb a bit --- docs/postgres.rst | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst index ca72131c6c..a04ab12b1a 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -3,7 +3,8 @@ Using Postgres Set up client ============= -We need to have installed the postgres python connector ``psycopg2``. In the + +Postgres support depends on the postgres python connector ``psycopg2``. In the virtual env:: sudo apt-get install libpq-dev @@ -13,7 +14,8 @@ virtual env:: Synapse config ============== -Add the following line to your config file:: +When you are ready to start using PostgreSQL, add the following line to your +config file:: database_config: @@ -37,6 +39,9 @@ adbapi connection pool. Porting from SQLite =================== +Overview +~~~~~~~~ + The script ``port_from_sqlite_to_postgres.py`` allows porting an existing synapse server backed by SQLite to using PostgreSQL. This is done in as a two phase process: @@ -44,12 +49,12 @@ phase process: 1. Copy the existing SQLite database to a separate location (while the server is down) and running the port script against that offline database. 2. Shut down the server. Rerun the port script to port any data that has come - in since taking the first snapshot. Restart server against the PostgrSQL + in since taking the first snapshot. Restart server against the PostgreSQL database. The port script is designed to be run repeatedly against newer snapshots of the SQLite database file. This makes it safe to repeat step 1 if there was a delay -between taking the previous snapshot and ready to do step 2. +between taking the previous snapshot and being ready to do step 2. It is safe to at any time kill the port script and restart it. @@ -57,7 +62,12 @@ Using the port script ~~~~~~~~~~~~~~~~~~~~~ Firstly, shut down the currently running synapse server and copy its database -file to another location. Once the copy is complete, restart synapse. +file (typically ``homeserver.db``) to another location. Once the copy is +complete, restart synapse. For instance:: + + ./synctl stop + cp homeserver.db homeserver.db.snapshot + ./synctl start Assuming your database config file (as described in the section *Synapse config*) is named ``database_config.yaml`` and the SQLite snapshot is at @@ -82,5 +92,6 @@ run:: --postgres-config database_config.yaml Once that has completed, change the synapse config to point at the PostgreSQL -database configuration file and restart synapse. Synapse should now be running -against PostgreSQL. +database configuration file using the ``database_config`` parameter (see +`Synapse Config`_) and restart synapse. Synapse should now be running against +PostgreSQL. -- cgit 1.4.1 From 72443572bf5dd147f50b0168e1078d88476a3e9f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Apr 2015 11:50:18 +0100 Subject: Mention that postgres databases must have the correct charset encoding --- docs/postgres.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst index a04ab12b1a..d645e1d697 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -1,6 +1,23 @@ Using Postgres -------------- +Set up database +=============== + +The PostgreSQL database used *must* have the correct encoding set, otherwise +would not be able to store UTF8 strings. To create a database with the correct +encoding use, e.g.:: + + CREATE DATABASE synapse + ENCODING 'UTF8' + LC_COLLATE='C' + LC_CTYPE='C' + template=template0 + OWNER synapse_user; + +This would create an appropriate database named ``synapse`` owned by the +``synapse_user`` user (which must already exist). + Set up client ============= -- cgit 1.4.1 From 32937f3ea03dd163bbcdc6f39efe22ffb760e0d6 Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Fri, 1 May 2015 14:06:43 +0100 Subject: database config is not kept in separate config file anymore --- docs/postgres.rst | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) (limited to 'docs') diff --git a/docs/postgres.rst b/docs/postgres.rst index d645e1d697..2dcc3caf9e 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -34,19 +34,15 @@ Synapse config When you are ready to start using PostgreSQL, add the following line to your config file:: - database_config: - -Where ```` is the file name that points to a yaml file of the -following form:: - - name: psycopg2 - args: - user: - password: - database: - host: - cp_min: 5 - cp_max: 10 + database: + name: psycopg2 + args: + user: + password: + database: + host: + cp_min: 5 + cp_max: 10 All key, values in ``args`` are passed to the ``psycopg2.connect(..)`` function, except keys beginning with ``cp_``, which are consumed by the twisted -- cgit 1.4.1 From 938939fd89518ba6db833c9463973f18d752fefc Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 6 May 2015 13:48:06 +0100 Subject: Move CAPTCHA_SETUP to docs/ --- CAPTCHA_SETUP | 31 ------------------------------- docs/CAPTCHA_SETUP | 31 +++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 31 deletions(-) delete mode 100644 CAPTCHA_SETUP create mode 100644 docs/CAPTCHA_SETUP (limited to 'docs') diff --git a/CAPTCHA_SETUP b/CAPTCHA_SETUP deleted file mode 100644 index 75ff80981b..0000000000 --- a/CAPTCHA_SETUP +++ /dev/null @@ -1,31 +0,0 @@ -Captcha can be enabled for this home server. This file explains how to do that. -The captcha mechanism used is Google's ReCaptcha. This requires API keys from Google. - -Getting keys ------------- -Requires a public/private key pair from: - -https://developers.google.com/recaptcha/ - - -Setting ReCaptcha Keys ----------------------- -The keys are a config option on the home server config. If they are not -visible, you can generate them via --generate-config. Set the following value: - - recaptcha_public_key: YOUR_PUBLIC_KEY - recaptcha_private_key: YOUR_PRIVATE_KEY - -In addition, you MUST enable captchas via: - - enable_registration_captcha: true - -Configuring IP used for auth ----------------------------- -The ReCaptcha API requires that the IP address of the user who solved the -captcha is sent. If the client is connecting through a proxy or load balancer, -it may be required to use the X-Forwarded-For (XFF) header instead of the origin -IP address. This can be configured as an option on the home server like so: - - captcha_ip_origin_is_x_forwarded: true - diff --git a/docs/CAPTCHA_SETUP b/docs/CAPTCHA_SETUP new file mode 100644 index 0000000000..75ff80981b --- /dev/null +++ b/docs/CAPTCHA_SETUP @@ -0,0 +1,31 @@ +Captcha can be enabled for this home server. This file explains how to do that. +The captcha mechanism used is Google's ReCaptcha. This requires API keys from Google. + +Getting keys +------------ +Requires a public/private key pair from: + +https://developers.google.com/recaptcha/ + + +Setting ReCaptcha Keys +---------------------- +The keys are a config option on the home server config. If they are not +visible, you can generate them via --generate-config. Set the following value: + + recaptcha_public_key: YOUR_PUBLIC_KEY + recaptcha_private_key: YOUR_PRIVATE_KEY + +In addition, you MUST enable captchas via: + + enable_registration_captcha: true + +Configuring IP used for auth +---------------------------- +The ReCaptcha API requires that the IP address of the user who solved the +captcha is sent. If the client is connecting through a proxy or load balancer, +it may be required to use the X-Forwarded-For (XFF) header instead of the origin +IP address. This can be configured as an option on the home server like so: + + captcha_ip_origin_is_x_forwarded: true + -- cgit 1.4.1 From 35698484a51081126d3f8b37a599d7d37a91f5fb Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 May 2015 18:51:09 +0100 Subject: Add some information on registering AS's --- CHANGES.rst | 2 ++ docs/application_services.rst | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 docs/application_services.rst (limited to 'docs') diff --git a/CHANGES.rst b/CHANGES.rst index dc9fcf98dd..f0bb973dcf 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -40,8 +40,10 @@ Application services: instead their configuration should be saved to a file and listed in the synapse ``app_service_config_files`` config option. The AS configuration file has the same format as the old ``/register`` request. + See `docs/application_services.rst`_ for more information. .. _`docs/postgres.rst`: docs/postgres.rst +.. _`docs/application_services.rst`: docs/application_services.rst .. _`Registration`: https://github.com/matrix-org/matrix-doc/blob/master/specification/10_client_server_api.rst#registration .. _`Retrieving Server Keys`: https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys .. _`Application Services`: https://github.com/matrix-org/matrix-doc/blob/0c6bd9/specification/25_application_service_api.rst#home-server---application-service-api diff --git a/docs/application_services.rst b/docs/application_services.rst new file mode 100644 index 0000000000..07236e62ae --- /dev/null +++ b/docs/application_services.rst @@ -0,0 +1,26 @@ +Registering an Application Service +================================== + +The registration of new application services is implementation dependent. In +synapse you need to create a new configuration file for you AS and add it to +the list of AS's specified under ``app_service_config_files`` synapse +config option. + +The format of the AS configuration file is as follows:: + + url: + as_token: + hs_token: + sender_localpart: + namespaces: + users: # List of users we're interested in + - exclusive: + regex: + - ... + aliases: [] # List of aliases we're interested in + rooms: [] # List of room ids we're interested in + +See the spec_ for further details on how application services work. + +.. _spec: https://github.com/matrix-org/matrix-doc/blob/master/specification/25_application_service_api.rst#application-service-api + -- cgit 1.4.1 From 5fe26a9b5ccabf68d833afe0021638bbb8deaa4d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 May 2015 18:54:53 +0100 Subject: Reword docs/application_services.rst --- docs/application_services.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) (limited to 'docs') diff --git a/docs/application_services.rst b/docs/application_services.rst index 07236e62ae..69f7e41b8a 100644 --- a/docs/application_services.rst +++ b/docs/application_services.rst @@ -1,12 +1,14 @@ Registering an Application Service ================================== -The registration of new application services is implementation dependent. In -synapse you need to create a new configuration file for you AS and add it to -the list of AS's specified under ``app_service_config_files`` synapse -config option. +The registration of new application services depends on the homeserver used. +In synapse you need to create a new configuration file for your AS and add it +to the list of AS's specified under the ``app_service_config_files`` config +option in your synapse config. -The format of the AS configuration file is as follows:: +The format of the AS configuration file is as follows: + +.. code-block:: yaml url: as_token: -- cgit 1.4.1 From 6101ce427a3e903449a1be6dbac4419714990298 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 7 May 2015 18:58:28 +0100 Subject: Slight rewording --- docs/application_services.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) (limited to 'docs') diff --git a/docs/application_services.rst b/docs/application_services.rst index 69f7e41b8a..a57bae6194 100644 --- a/docs/application_services.rst +++ b/docs/application_services.rst @@ -2,10 +2,18 @@ Registering an Application Service ================================== The registration of new application services depends on the homeserver used. -In synapse you need to create a new configuration file for your AS and add it -to the list of AS's specified under the ``app_service_config_files`` config +In synapse, you need to create a new configuration file for your AS and add it +to the list specified under the ``app_service_config_files`` config option in your synapse config. +For example: + +.. code-block:: yaml + + app_service_config_files: + - /home/matrix/.synapse/.yaml + + The format of the AS configuration file is as follows: .. code-block:: yaml -- cgit 1.4.1