From 56f518d279b642efce92e172b463232937d50b8c Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 27 Apr 2015 14:53:35 +0100 Subject: Add docs on how to use synapse with psycopg2 --- docs/postgres.rst | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 docs/postgres.rst (limited to 'docs/postgres.rst') diff --git a/docs/postgres.rst b/docs/postgres.rst new file mode 100644 index 0000000000..5bb45d3ed9 --- /dev/null +++ b/docs/postgres.rst @@ -0,0 +1,34 @@ +Using Postgres +-------------- + +Set up client +============= +We need to have installed the postgres python connector ``psycopg2``. In the +virtual env:: + + sudo apt-get install libpq-dev + pip install psycopg2 + + +Synapse config +============== + +Add the following line to your config file:: + + database_config: + +Where ```` is the file name that points to a yaml file of the +following form:: + + name: psycopg2 + args: + user: + password: + database: + host: + cp_min: 5 + cp_max: 10 + +All key, values in ``args`` are passed to the ``psycopg2.connect(..)`` +function, except keys beginning with ``cp_``, which are consumed by the twisted +adbapi connection pool. -- cgit 1.5.1 From 3151afee9e3213ba439f701e9b968a32ec41054d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Apr 2015 17:59:27 +0100 Subject: Update docs/postgres.rst to explain port script usage --- docs/postgres.rst | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) (limited to 'docs/postgres.rst') diff --git a/docs/postgres.rst b/docs/postgres.rst index 5bb45d3ed9..bbf1b12a89 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -32,3 +32,54 @@ following form:: All key, values in ``args`` are passed to the ``psycopg2.connect(..)`` function, except keys beginning with ``cp_``, which are consumed by the twisted adbapi connection pool. + + +Porting from SQLite +=================== + +The script ``port_from_sqlite_to_postgres.py`` allows porting an existing +synapse server backed by SQLite to using PostgreSQL. This is done in as a two +phase process: + +1. Copy the existing SQLite database to a separate location (while the server + is down) and running the port script against that offline database. +2. Shut down the server. Rerun the port script to port any data that has come + in since taking the first snapshot. Restart server against the PostgrSQL + database. + +The port script is designed to be run repeatedly against newer snapshots of the +SQLite database file. This makes it safe to repeat step 1 if there was a delay +between taking the previous snapshot and ready to do step 2. + +It is safe to at any time kill the port script and restart it. + +Using the port script +~~~~~~~~~~~~~~~~~~~~~ + +Firstly, shut down the currently running synapse server and copy its database +file to another location. Once the copy is complete, restart synapse. + +Assuming your database config file (as described in the section *Synapse +config*) is named ``database_config.yaml`` and the SQLite snapshot is at +``homeserver.db.snapshot`` then simply run:: + + python scripts/port_from_sqlite_to_postgres.py \ + --sqlite-database homeserver.db.snapshot \ + --postgres-config database_config.yaml + +The flag ``--curses`` displays a coloured curses progress UI. + +If the script took a long time to complete, or time has otherwise passed since +the original snapshot was taken, repeat the previous steps with a newer +snapshot. + +To complete the conversion shut down the synapse server and run the port +script one last time, e.g. if the SQLite database is at ``homeserver.db`` run: + + python scripts/port_from_sqlite_to_postgres.py \ + --sqlite-database homeserver.db \ + --postgres-config database_config.yaml + +Once that has completed, change the synapse config to point at the PostgreSQL +database configuration file and restart synapse. Synapse should now be running +against PostgreSQL. -- cgit 1.5.1 From cc52f02d74e5eee4eb64ff16fb0dfaa7d4d7d38e Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Apr 2015 18:09:20 +0100 Subject: Fix rst --- docs/postgres.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'docs/postgres.rst') diff --git a/docs/postgres.rst b/docs/postgres.rst index bbf1b12a89..ca72131c6c 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -74,7 +74,8 @@ the original snapshot was taken, repeat the previous steps with a newer snapshot. To complete the conversion shut down the synapse server and run the port -script one last time, e.g. if the SQLite database is at ``homeserver.db`` run: +script one last time, e.g. if the SQLite database is at ``homeserver.db`` +run:: python scripts/port_from_sqlite_to_postgres.py \ --sqlite-database homeserver.db \ -- cgit 1.5.1 From 478e511db04548023f5a652e6ed7117ecd3d3cf5 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Wed, 29 Apr 2015 00:48:07 +0100 Subject: improve postgres blurb a bit --- docs/postgres.rst | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) (limited to 'docs/postgres.rst') diff --git a/docs/postgres.rst b/docs/postgres.rst index ca72131c6c..a04ab12b1a 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -3,7 +3,8 @@ Using Postgres Set up client ============= -We need to have installed the postgres python connector ``psycopg2``. In the + +Postgres support depends on the postgres python connector ``psycopg2``. In the virtual env:: sudo apt-get install libpq-dev @@ -13,7 +14,8 @@ virtual env:: Synapse config ============== -Add the following line to your config file:: +When you are ready to start using PostgreSQL, add the following line to your +config file:: database_config: @@ -37,6 +39,9 @@ adbapi connection pool. Porting from SQLite =================== +Overview +~~~~~~~~ + The script ``port_from_sqlite_to_postgres.py`` allows porting an existing synapse server backed by SQLite to using PostgreSQL. This is done in as a two phase process: @@ -44,12 +49,12 @@ phase process: 1. Copy the existing SQLite database to a separate location (while the server is down) and running the port script against that offline database. 2. Shut down the server. Rerun the port script to port any data that has come - in since taking the first snapshot. Restart server against the PostgrSQL + in since taking the first snapshot. Restart server against the PostgreSQL database. The port script is designed to be run repeatedly against newer snapshots of the SQLite database file. This makes it safe to repeat step 1 if there was a delay -between taking the previous snapshot and ready to do step 2. +between taking the previous snapshot and being ready to do step 2. It is safe to at any time kill the port script and restart it. @@ -57,7 +62,12 @@ Using the port script ~~~~~~~~~~~~~~~~~~~~~ Firstly, shut down the currently running synapse server and copy its database -file to another location. Once the copy is complete, restart synapse. +file (typically ``homeserver.db``) to another location. Once the copy is +complete, restart synapse. For instance:: + + ./synctl stop + cp homeserver.db homeserver.db.snapshot + ./synctl start Assuming your database config file (as described in the section *Synapse config*) is named ``database_config.yaml`` and the SQLite snapshot is at @@ -82,5 +92,6 @@ run:: --postgres-config database_config.yaml Once that has completed, change the synapse config to point at the PostgreSQL -database configuration file and restart synapse. Synapse should now be running -against PostgreSQL. +database configuration file using the ``database_config`` parameter (see +`Synapse Config`_) and restart synapse. Synapse should now be running against +PostgreSQL. -- cgit 1.5.1 From 72443572bf5dd147f50b0168e1078d88476a3e9f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Apr 2015 11:50:18 +0100 Subject: Mention that postgres databases must have the correct charset encoding --- docs/postgres.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) (limited to 'docs/postgres.rst') diff --git a/docs/postgres.rst b/docs/postgres.rst index a04ab12b1a..d645e1d697 100644 --- a/docs/postgres.rst +++ b/docs/postgres.rst @@ -1,6 +1,23 @@ Using Postgres -------------- +Set up database +=============== + +The PostgreSQL database used *must* have the correct encoding set, otherwise +would not be able to store UTF8 strings. To create a database with the correct +encoding use, e.g.:: + + CREATE DATABASE synapse + ENCODING 'UTF8' + LC_COLLATE='C' + LC_CTYPE='C' + template=template0 + OWNER synapse_user; + +This would create an appropriate database named ``synapse`` owned by the +``synapse_user`` user (which must already exist). + Set up client ============= -- cgit 1.5.1