MariaDB Server and MariaDB Cluster are open source products powered by the MariaDB Corporation. MariaDB Server is one of the most popular relational databases, it was originally forked from MySQL server.

MariaDB Cluster is a high availability solution built from MariaDB Server, using a Galera Cluster wsrep library to synchronize the data between nodes.  The replication method of Galera is synchronous (or ‘virtually synchronous’), ensuring the data will be the same on all the nodes.

MariaDB server can also be made highly available via standard replication. Replication can be asynchronous or semi-synchronous. 

So how does the MariaDB server with standard replication differ from MariaDB Cluster with Galera Cluster? In this blog, we will compare those two. We will be using ClusterControl to illustrate some of the differences. 

MariaDB Server Architecture

The architecture of MariaDB Server can be a single/standalone instance or master/slave replication as shown in the diagram below. 

The MariaDB Server single instance architecture stands for one node only. The drawback of having a single instance is a single point of failure for the database. If your database crashes and does not come back up, you do not have any failover mechanism, and you need to do a restore to recover your database from the last backup. 

The master/slave architecture is a distributed setup, with the master acting as writer and the slave(s) as reader(s). Using a load balancer like Maxscale or ProxySQL, you can split the database traffic so that writes are sent to the master and reads to the slave(s). Having a replication setup will eliminate a single point of failure for the database, but you need to be able to failover automatically if the master fails. Or else, applications will not be able to write to the database and they will be affected. ClusterControl can be configured to provide automatic failover and recovery for MariaDB replication.

MariaDB Cluster Architecture

MariaDB Cluster is a high availability solution consisting of MariaDB Server and Galera Replication as shown in the architecture diagram below :

It is synchronous (“virtually synchronous”) replication, all of the nodes are writable. The synchronous replication guarantees if  the changes happen in one of the galera nodes, it will be available on all the other nodes in the cluster before being committed.

The big difference is that all the nodes are equal from the application point of view, they can send write traffic to any of the database instances. Also, all nodes should have exactly the same data so there is no data loss in case of node failure. 

MariaDB Deployment

Both MariaDB Replication and MariaDB Cluster can be deployed via ClusterControl. When you deploy MariaDB Server, you need to start by choosing MySQL Replication while for MariaDB Cluster, you need to choose MySQL Galera. 

For MariaDB Server, you can either deploy a single node MariaDB instance or you can setup master/slave and bi-directional replication. The minimum number of nodes in a replication setup is two, you need one master and at least one slave.Just fill the IP Address for the master and add slaves (if you want to have master/slave architecture). You can use the Add Second Master field if you want to set up bi-directional replication. A master-master setup will be provisioned with bi-directional replication, but one of the nodes will be set as read-only. The reason is to minimize the risk for data drift and ‘errant transactions’. 

For MariaDB Cluster, you need at least 3 hosts for target database nodes to be installed. This is because it has to be able to handle network partitioning or “split brain” syndrome.  You just need to fill the ip address when Add Node when defining MySQL Servers configuration.

Do not forget to choose MariaDB as the vendor of database, database version that you want to install and fill the root password. You can also change the non default datadir to any other path. 

After we configure all of the things, just deploy the cluster. It will trigger a new job for database deployment.

Note that it is also possible to have 2 Galera nodes and one Galera arbitrator aka garbd on a third host.  

MariaDB Server & Cluster Monitoring

Database monitoring is a critical part of the database, you can know the current state of database health. The difference between MariaDB Server and MariaDB Cluster monitoring is Galera Metrics for synchronization.

On MariaDB Server, you can check your current database health through the MySQL Metrics; MySQL Server - General, MySQL Server - Caches, MySQL InnoDB Metrics which is also visible on the MariaDB Cluster as shown in below:

MySQL Server- General gives you information about the current state of InnoDB buffer pool hit ratio, database connection, queries, locking, and database memory utilization.

MySQL Server- Caches, there is a lot of information provided in Caches. Mostly related to the caching in database, eg: buffer pool size, buffer pool instance. There is also information about table cache usage, hit ratio, Cache Hits and Misses. You can also find thread cache usage and hit ratio information .

MySQL Server- InnoDB Metrics shows metrics related to InnoDB storage eg : Bufferpool activity, InnoDB Row operations, InnoDB Log file size, InnoDB Data Read/Write.

On MariaDB Server, if you setup master/slave replication, there is one subcategory of metrics under MySQL Replication - Master. There is information related to master binary log file, master binary log position, and binlog creation frequency. 

MariaDB Server has a lot of information related to the database, these are also available for MariaDB Cluster. The difference is there are two dashboards for MariaDB Cluster - Galera Overview and Galera Server Charts.

Galera Overview gives information related to the current state of Galera replication. There is information like cluster size, flow control sent, flow control received, flow control paused.

Galera Server Charts has information about cluster name, cluster status, size, Global cache size. 

Conclusion

MariaDB Server with standard replication and MariaDB Cluster are not really different products in terms of database service, but they have different characteristics depending on your requirements on availability and scalability. ClusterControl supports both MariaDB Server with standard replication and MariaDB Cluster deployments, so do give both setups a try and let us know your thoughts.

Moodle is an open-source e-learning platform (aka Learning Management System) that is widely adopted by educational institutions to create and administer online courses. For larger student bodies and higher volumes of instruction, moodle must be robust enough to serve thousands of learners, administrators, content builders and instructors simultaneously.

Availability and scalability are key requirements as moodle becomes a critical application for course providers. In this blog, we will show you how to deploy and cluster moodle/web, database and file-system components on multiple servers to achieve both high availability and scalability.

We are going to deploy Moodle 3.9 LTS on top of the GlusterFS clustered file system and MariaDB Cluster 10.4. To eliminate any single point of failure, we will use three nodes to serve the application and database while the remaining two are used for load balancers and the ClusterControl management server. This is illustrated in the following diagram:

Prerequisites

All hosts are running on CentOS 7.6 64bit with SElinux and iptables disabled. The following is the host definition inside /etc/hosts:

192.168.30.100   moodle web db virtual-ip mysql
192.168.30.101   moodle1 web1 db1
192.168.30.102   moodle2 web2 db2
192.168.30.103   moodle3 web3 db3
192.168.30.111   lb1
192.168.30.112   lb2 clustercontrol

Creating Gluster replicated volumes on the root partition is not recommended. We will use another disk /dev/sdb1 on every web node (web1, web2 and web3) for the GlusterFS storage backend, mounted as /storage. This is shown with the following mount command:

$ mount | grep storage
/dev/sdb1 on /storage type ext4 (rw,relatime,seclabel,data=ordered)

Moodle Database Cluster Deployment

We will start with database deployment for our three-node MariaDB Cluster using ClusterControl. Install ClusterControl on lb2:

$ wget https://severalnines.com/downloads/cmon/install-cc
$ chmod 755 install-cc
$ ./install-cc

Then set up passwordless SSH to all nodes that are going to be managed by ClusterControl (3 MariaDB Cluster + 2 HAProxy nodes). On the ClusterControl node, do:

$ whoami
root
$ ssh-keygen -t rsa # Press Enter on all prompts
$ ssh-copy-id 192.168.30.111 #lb1
$ ssh-copy-id 192.168.30.112 #lb2
$ ssh-copy-id 192.168.30.101 #moodle1
$ ssh-copy-id 192.168.30.102 #moodle2
$ ssh-copy-id 192.168.30.103 #moodle3

**Enter the root password for the respective host when prompted.

Open a web browser and go to https://192.168.30.102/clustercontrol and create an admin user. Then go to Deploy -> MySQL Galera. Follow the deployment wizard accordingly. At the second stage 'Define MySQL Servers', pick MariaDB 10.4 and specify the IP address for every database node. Make sure you get a green tick after entering the database node details, as shown below:

Click "Deploy" to start the deployment. The database cluster will be ready in 15~20 minutes. You can follow the deployment progress at Activity -> Jobs -> Create Cluster -> Full Job Details. The cluster will be listed under the Database Cluster dashboard once deployed.

We can now proceed to database load balancer deployment.

Load Balancers and Virtual IP for Moodle

Since HAProxy and ClusterControl are co-located on the same server (lb2), we need to change the Apache default port to another port, for example, port 8080 so it won't conflict with port 80 that is going to be assigned to HAProxy for Moodle web load balancing.

On lb2, open the Apache configuration file at /etc/httpd/conf/httpd.conf and make the following changes:

Listen 8080

Also we need to make changes on the ClusterControl's virtualhost definition and modify the VirtualHost line inside /etc/httpd/conf.d/s9s.conf:

<VirtualHost *:8080>

Restart Apache webserver to apply changes:

$ systemctl restart httpd

At this point, ClusterControl is accessible via port 8080 on HTTP at http://192.168.30.112:8080/clustercontrol or you can connect through the default HTTPS port at https://192.168.30.112/clustercontrol.

Login to ClusterControl, drill down to the database cluster and go to Add Load Balancer -> HAProxy -> Deploy HAProxy. Deploy lb1 and lb2 with HAProxy similar to the screenshots below:

Note that we are going to set up 3 listening ports for HAProxy, 3307 for database single-writer (deployed via ClusterControl), 3308 for database multi-writer/reader (will be customized later on) and port 80 for Moodle web load balancing (will be customized later on).

Repeat the same deployment step for lb2 (192.168.30.112). 

Modify the HAProxy configuration files on both HAProxy instances (lb1 and lb2) by adding the following lines for database multi-writer (3308) and Moodle load balancing (80):

listen  haproxy_192.168.30.112_3308_ro
        bind *:3308
        mode tcp
        tcp-check connect port 9200
        timeout client  10800s
        timeout server  10800s
        balance leastconn
        option httpchk
        default-server port 9200 inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 64 maxqueue 128 weight 100
        server 192.168.30.101 192.168.30.101:3306 check
        server 192.168.30.102 192.168.30.102:3306 check
        server 192.168.30.103 192.168.30.103:3306 check

frontend http-in
        bind *:80
        default_backend web_farm

backend web_farm
        server moodle1 192.168.30.101:80 maxconn 32 check
        server moodle2 192.168.30.102:80 maxconn 32 check
        server moodle3 192.168.30.103:80 maxconn 32 check

Restart the HAProxy service to load the changes:

$ systemctl restart haproxy

At this point, we should be able to see the HAProxy stats page (accessible under ClusterControl -> Nodes -> pick the HAProxy nodes) reporting as below:

The first section (with 2 blue lines) represents the single-writer configuration on port 3307. All database connections coming to this port will be forwarded to only one database server (the green line). The second section represents the multi-writer configuration on port 3308, where all database connections will be forwarded to multiple available nodes using leastconn balancing algorithm. The last section in red represents the Moodle web load balancing that we are yet to set up. 

Go to ClusterControl -> Manage -> Load Balancer -> Keepalived -> Deploy Keepalived -> HAProxy and pick both HAProxy instances that will be tied together with a virtual IP address:

8. The load balancer nodes have now been installed, and are integrated with ClusterControl. You can verify this by checking out the ClusterControl’s summary bar:

Deploying GlusterFS

*The following steps should be performed on moodle1, moodle2 and moodle3 unless specified otherwise.

Install GlusterFS repository for CentOS 7 and install GlusterFS server and all of its dependencies:

$ yum install centos-release-gluster -y
$ yum install glusterfs-server -y

Create a directory called brick under /storage partition:

$ mkdir /storage/brick

Start GlusterFS service:

$ systemctl start glusterd

On moodle1, probe the other nodes (moodle2 and moodle3):

$ gluster peer probe 192.168.30.102
peer probe: success.

$ gluster peer probe 192.168.30.103
peer probe: success.

You can verify the peer status with the following command:

$ gluster peer status
Number of Peers: 2

Hostname: 192.168.30.102
Uuid: dd976496-fda4-4ace-801e-7ee55461b458
State: Peer in Cluster (Connected)

Hostname: 192.168.30.103
Uuid: 27a81fba-c529-4c2c-9cf7-2dcaf5880cef
State: Peer in Cluster (Connected)

On moodle1, create a replicated volume on probed nodes:

$ gluster volume create rep-volume replica 3 192.168.30.101:/storage/brick 192.168.30.102:/storage/brick 192.168.30.103:/storage/brick
volume create: rep-volume: success: please start the volume to access data

Start the replicated volume on moodle1:

$ gluster volume start rep-volume
volume start: rep-volume: success

Ensure the replicated volume and processes are online:

$ gluster volume status
Status of volume: rep-volume

Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 192.168.30.101:/storage/brick         49152     0          Y       15059
Brick 192.168.30.102:/storage/brick         49152     0          Y       24916
Brick 192.168.30.103:/storage/brick         49152     0          Y       23182
Self-heal Daemon on localhost               N/A       N/A        Y       15080
Self-heal Daemon on 192.168.30.102          N/A       N/A        Y       24944
Self-heal Daemon on 192.168.30.103          N/A       N/A        Y       23203

Task Status of Volume rep-volume
------------------------------------------------------------------------------
There are no active volume tasks

Moodle requires a data directory (moodledata) to be outside of the webroot (/var/www/html). So, we’ll mount the replicated volume on /var/www/moodledata instead. Create the path:

$ mkdir -p /var/www/moodledata

Add following line into /etc/fstab to allow auto-mount:

localhost:/rep-volume /var/www/moodledata glusterfs defaults,_netdev 0 0

Mount the GlusterFS volume into /var/www/moodledata:

$ mount -a

Apache and PHP Setup for Moodle

*The following steps should be performed on moodle1, moodle2 and moodle3.

Moodle 3.9 requires PHP 7.4. To simplify the installation, install the Remi YUM repository:

$ yum -y install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm

Install all required packages with remi repository enabled:

$ yum-config-manager --enable remi-php74
$ yum install -y httpd php php-xml php-gd php-mbstring php-soap php-intl php-mysqlnd php-pecl-zendopcache php-xmlrpc php-pecl-zip

Start Apache web server and enable it on boot:

$ systemctl start httpd
$ systemctl enable httpd

Installing Moodle

*The following steps should be performed on moodle1.

Download Moodle 3.9 from download.moodle.org and extract it under /var/www/html:

$ wget http://download.moodle.org/download.php/direct/stable39/moodle-latest-39.tgz
$ tar -xzf moodle-latest-39.tgz -C /var/www/html

Moodle requires a data directory outside /var/www/html. The default location would be /var/www/moodledata. Create the directory and assign correct permissions:

$ mkdir /var/www/moodledata
$ chown apache.apache -R /var/www/html /var/www/moodledata

Before we proceed with the Moodle installation, we need to prepare the Moodle database. From ClusterControl, go to Manage -> Schema and Users -> Create Database and create a database called ‘moodle’.

Create a new database user called "moodle" under the Users tab. Allow this user to be accessed from the 192.168.30.0/24 IP address range, with ALL PRIVILEGES on the moodle database:

Open the browser and navigate to http://192.168.30.101/moodle/. You should see moodle’s installation page. Accept all default values and choose "MariaDB (native/mariadb)" in the database driver dropdown. Then, proceed to the "Database settings" page. Enter the configuration details as below:

The "Database host" value is the virtual IP address that we have created with Keepalived, while the database port, is the single-writer MySQL port load-balanced by HAProxy.

Proceed to the last page and let the Moodle installation process complete. After completion, you should see the Moodle admin dashboard.

Moodle 3.9 supports read-only database instances, where we can use the multi-writer load-balanced port 3308 to distribute read-only connections from Moodle to multiple servers. Add the "readonly" dboptions array inside config.php as the following example:

<?php  // Moodle configuration file

unset($CFG);
global $CFG;
$CFG = new stdClass();

$CFG->dbtype    = 'mariadb';
$CFG->dblibrary = 'native';
$CFG->dbhost    = '192.168.30.100';
$CFG->dbname    = 'moodle';
$CFG->dbuser    = 'moodle';
$CFG->dbpass    = 'password';
$CFG->prefix    = 'mdl_';
$CFG->dboptions = array (
  'dbpersist' => 0,
  'dbport' => 3307,
  'dbsocket' => '',
  'dbcollation' => 'utf8mb4_general_ci',
  'readonly' => [
    'instance' => [
      'dbhost' => '192.168.30.100',
      'dbport' => 3308,
      'dbuser' => 'moodle',
      'dbpass' => 'password'
    ]
  ]
);

$CFG->wwwroot   = 'http://192.168.30.100/moodle';
$CFG->dataroot  = '/var/www/moodledata';
$CFG->admin     = 'admin';

$CFG->directorypermissions = 0777;

require_once(__DIR__ . '/lib/setup.php');

In the configuration above, we also modified the $CFG->wwwroot line and change the IP address to the virtual IP address, which will be load-balanced by HAProxy on port 80.

Once the above is configured, we can copy the content of /var/www/html/moodle from moodle1 to moodle2 and moodle3, so Moodle can be served by the other web servers as well. Once done, you should be able to access the load-balanced Moodle on http://192.168.30.100/moodle.

Check the HAProxy statistics by logging into ClusterControl -> Nodes -> pick the primary HAProxy node. You should see some statistics bytes in and out on both 3307_rw and 3308_ro sections:

This indicates that the Moodle reads/writes database workloads are distributed among the backend MariaDB Cluster servers.

Congratulations, you have now deployed a scalable Moodle infrastructure with clustering on the web, database and file system layers.  To download ClusterControl, click here.

Face-to-face meetings, nowadays, are limited to the bare minimum, online activities have taken over as the main way for teacher - student interaction. It increased the stress on the existing online “meeting” platforms (is there anyone who does not know what Zoom is nowadays?) but also on online learning platforms. The high availability of the online tools is more important than ever and the operation teams rush to build durable, highly available architectures for their environments.

Most likely at least some of you have used Moodle - it is a standalone online learning platform that you can deploy on premises and use it to deliver online training for your organization. As we mentioned, it is as important as ever to make it work in a durable, highly available fashion. We would like to propose a highly available solution that involves MariaDB as a backend database - both asynchronous replication and Galera Cluster.

Environment Design Process

We would like to start with a process where we would explain the thought process behind designing the environment for Moodle. We want high availability therefore a single database node does not work for us. We want multiple nodes and this leads us to the first design decision. Should we use asynchronous replication or Galera Cluster? Second question is: how will we distribute the workload across the nodes? Let’s start with the second one.

The latest Moodle version at the time when this blog has been written (3.9) introduced a nice feature called safe reads. The problem to solve here is read after write. When you use one node, the world is a simple place. You write and then you read. Whatever you wrote is there already. When you add nodes, though, things change. In asynchronous replication slaves may be lagging behind even tens of seconds or more. Whatever you write on the master may take even minutes (if not more in the more extreme cases) to be applied to the slave. If you execute a write and then immediately attempt to read the same data from one of the slaves, you may be up to a nasty surprise - data will not be there. Galera cluster uses a “virtually” synchronous replication and in this particular case “virtually” makes a huge difference - Galera is not immune to the read-after-write problems. There’s always delay between write execution on the local node and writeset being applied to remaining nodes of the cluster. Sure, it’s most likely measured in milliseconds rather than seconds but it still may break the assumption that you can immediately read what you wrote. The only place where you can safely read after writing is the node on which you wrote the data.

As Moodle relies on the read-after-write quite a lot, we cannot easily scale reads only by adding more nodes to read from. For Galera Cluster we could attempt to mitigate the issue by using wsrep-sync-wait configuration setting to force Galera to ensure that the reads are safe to execute. This creates the performance impact on the system as all reads have to wait for writes to be applied before they can be executed. This is also a solution for MariaDB Cluster (and other Galera-based solutions), not for asynchronous replication. Luckily, the solution from Moodle solves this issue. You can define a list of nodes that may possibly be lagging and Moodle will use them only for reads that do not require to be up to date with the writes. All remaining reads that require data to be always up to date would be directed to the writer node.So, Moodle’s scalability is sort of limited as only the “safe” reads can be scaled out. We will definitely want to use the 3.9’s feature given that this is the only safe method to determine which select should go where. Given that everything is written in a Moodle’s configuration file, we would most likely want to use a load balancer, preferably ProxySQL, to create logic that would handle our read distribution.

Should we use MariaDB Cluster or asynchronous replication? We will actually show you how to use both. In both cases the configuration for the Moodle will be pretty much the same. In both cases we will utilize ProxySQL as the loadbalancer. The main difference between those solutions is the failover. MariaDB Cluster is way easier to deal with - if one node is down, ProxySQL will simply move the write traffic to one of the remaining nodes. With asynchronous replication things are slightly different though. If the master goes down, failover has to happen. This does not happen automatically, you either have to perform it by hand or you can rely on some software to accomplish that. In our case we will use ClusterControl to manage the environment and perform the failover therefore, from the user standpoint, there’s not much of a difference between asynchronous replication and the MariaDB Cluster - in both cases writer failure will be automatically handled and cluster will automatically recover.

What we have established is that we will showcase both asynchronous and virtually synchronous replication. We will use the safe writes feature from Moodle 3.9 and we will use ProxySQL as the loadbalancer. To ensure high availability we will need more than one ProxySQL instance therefore we will go with two of them and to create a single point of entry into the database layer we will use Keepalived to create a Virtual IP and point it to one of the available ProxySQL nodes. Here’s how our database cluster may look like:

For asynchronous replication this could look something like this:

Deploying a Highly Available Database Backend for Moodle Using MariaDB Replication

Let’s start with the MariaDB Replication. We are going to use ClusterControl to deploy the whole database backend including load balancers.

Deploying MariaDB Replication Cluster

At first, we need to pick “Deploy” from the wizard:

Then we should define SSH connectivity, passwordless, key-based SSH access is a requirement for ClusterControl to manage database infrastructure.

When you fill those details, it’s time to pick a vendor and a version, define superuser’s password and decide on some other details.

We are going to use MariaDB 10.4 for now. As a next step we have to define the replication topology:

We should pass the hostnames of the nodes and how they should relate to each other. Once we are happy with the topology, we can deploy. For the purpose of this blog we will use master and two slaves as our backend.

We have our first cluster up and ready. Now, let’s deploy ProxySQL and Keepalived.

Deploying ProxySQL

For ProxySQL it’s required to fill in some details - pick the host to install it on, decide on ProxySQL version, credentials for the administrative and monitoring users. You should also import existing database users or create a new one for your application. Finally, decide which database nodes you want to use with ProxySQL and decide if you use implicit transactions. In the case of Moodle this is not true.

Deploying Keepalived

As the next step we will deploy Keepalived.

After passing details like ProxySQL instances that should be monitored, Virtual IP and the interface VIP should bind to we are ready to deploy. After couple of minutes everything should be ready and the topology should look like below:

Configure Moodle and ProxySQL for Safe Writes Scale-Out

The final step will be to configure Moodle and ProxySQL to use safe writes. While it is possible to hardcode database nodes in the Moodle configuration, it would be much better to rely on ProxySQL to handle the topology changes. What we can do is to create an additional user in the database. That user will be configured in Moodle to execute safe reads. ProxySQL will be configured to send all traffic executed from that user to the available slave nodes.

First, let’s create a user that we’ll use for read-only access.

We are granting all privileges here but it should be possible to limit that list.

User that we just created has to be added to both ProxySQL instances that we have in the cluster in order to allow ProxySQL to authenticate as that user. In the ClusterControl UI you can use the “Import User” action.

We can search for the user that we just created:

ProxySQL uses a concept of hostgroups - groups of hosts that serve the same purpose. In our default configuration there are two hostgroups - hostgroup 10 which always point to current master and hostgroup 20 which points towards slave nodes. We want this user to send the traffic to slave nodes therefore we will assign HG 20 as the default one.

That’s it, the user will be shown on the list of the users:

Now we should repeat the same process on the other ProxySQL node or use the “Sync Instances” option. One way or the other, both ProxySQL nodes should have the moodle_safereads user added.

The last step will be to deploy Moodle. We won’t go here through the whole process, but there’s one issue we have to address. ProxySQL presents itself as 5.5.30 and Moodle complains it is too old. We can edit it easily to whatever version we want:

Once this is done, we have to temporarily send all of the traffic to the master. This can be accomplished by deleting all of the query rules in ProxySQL. The ‘moodle’ user has HG10 as the default hostgroup which means that with no query rules all traffic from that user will be directed to the master. The second, safe reads, user has the default hostgroup 20 which is pretty much all the configuration we want to have in place.

Once this is done, we should edit Moodle’s configuration file and enable the safe reads feature:

<?php  // Moodle configuration file



unset($CFG);

global $CFG;

$CFG = new stdClass();



$CFG->dbtype    = 'mysqli';

$CFG->dblibrary = 'native';

$CFG->dbhost    = '192.168.1.111';

$CFG->dbname    = 'moodle';

$CFG->dbuser    = 'moodle';

$CFG->dbpass    = 'pass';

$CFG->prefix    = 'mdl_';

$CFG->dboptions = array (

  'dbpersist' => 0,

  'dbport' => 6033,

  'dbsocket' => '',

  'dbcollation' => 'utf8mb4_general_ci',

  'readonly' => [

    'instance' => [

      'dbhost' => '192.168.1.111',

      'dbport' => 6033,

      'dbuser' => 'moodle_safereads',

      'dbpass' => 'pass'

    ]

  ]



);



$CFG->wwwroot   = 'http://192.168.1.200/moodle';

$CFG->dataroot  = '/var/www/moodledata';

$CFG->admin     = 'admin';



$CFG->directorypermissions = 0777;



require_once(__DIR__ . '/lib/setup.php');



// There is no php closing tag in this file,

// it is intentional because it prevents trailing whitespace problems!

What happened here is that we added the read only connection to ProxySQL which will use moodle_safereads user. This user will always point towards slaves. This concludes our setup of Moodle for MariaDB replication.

Deploying a Highly Available Database Backend for Moodle Using MariaDB Cluster

This time we’ll try to use MariaDB Cluster as our backend. Again, the first step is the same, we need to pick “Deploy” from the wizard:

Once you do that, we should define SSH connectivity, passwordless, key-based SSH access is a requirement for ClusterControl to manage database infrastructure.

Then we should decide on the vendor, version, password hosts and couple more settings:

Once we fill all the details, we are good to deploy.

We could continue here further but given all further steps are basically the same as with MariaDB replication, we would just ask you to scroll up and check the “Deploying ProxySQL” section and everything that follows it. You have to deploy ProxySQL, Keepalived, reconfigure it, change Moodle’s configuration file and this is pretty much it. We hope that this blog will help you to build highly available environments for Moodle backed by MariaDB Cluster or replication.

Moodle is a very popular platform to run online courses. With the situation we see in 2020, Moodle, along with communicators like Zoom forms the backbone of the services that allow online learning and stay-at-home education. The demand put on Moodle platforms significantly increased compared to previous years. New platforms have been built, additional load has been put on the platforms that, historically, only acted as a helper tool and now they are intended to drive the whole educational effort. How to scale out the Moodle? We have a blog on this topic. How to scale the database backend for Moodle? Well, that’s another story. Let’s take a look at it as scaling out databases is not the easiest thing to do, especially if the Moodle adds its own little twist.

As the entry point we will use the architecture described in one of our earlier posts. MariaDB Cluster with ProxySQL and Keepalived on top of things. 

As you can see, we have a three node MariaDB Cluster with ProxySQL that splits safe reads from the rest of the traffic based on the user. 

<?php  // Moodle configuration file



unset($CFG);

global $CFG;

$CFG = new stdClass();



$CFG->dbtype    = 'mysqli';

$CFG->dblibrary = 'native';

$CFG->dbhost    = '192.168.1.222';

$CFG->dbname    = 'moodle';

$CFG->dbuser    = 'moodle';

$CFG->dbpass    = 'pass';

$CFG->prefix    = 'mdl_';

$CFG->dboptions = array (

  'dbpersist' => 0,

  'dbport' => 6033,

  'dbsocket' => '',

  'dbcollation' => 'utf8mb4_general_ci',

  'readonly' => [

    'instance' => [

      'dbhost' => '192.168.1.222',

      'dbport' => 6033,

      'dbuser' => 'moodle_safereads',

      'dbpass' => 'pass'

    ]

  ]



);



$CFG->wwwroot   = 'http://192.168.1.200/moodle';

$CFG->dataroot  = '/var/www/moodledata';

$CFG->admin     = 'admin';



$CFG->directorypermissions = 0777;



require_once(__DIR__ . '/lib/setup.php');



// There is no php closing tag in this file,

// it is intentional because it prevents trailing whitespace problems!

User, as shown above, is defined in the Moodle configuration file. This allows us to automatically and safely send writes and all SELECT statements that require data consistency to the writer node while still sending some of the SELECTs to the remaining nodes in the MariaDB Cluster.

Let’s assume that this particular setup is not enough for us. What are the options that we have? We have two main elements in the setup - MariaDB Cluster and ProxySQL. We’ll consider issues on both sides:

• What can be done if the ProxySQL instance cannot cope with traffic?
• What can be done if MariaDB Cluster cannot cope with traffic?

Let’s start with the first scenario.

ProxySQL Instance is Overloaded

In the current environment only one ProxySQL instance can be handling the traffic - the one that Virtual IP points to. This leaves us with a ProxySQL instance that is acting as a standby - up and running but not used for anything. If the active ProxySQL instance is getting close to CPU saturation, there are a couple of things you may want to do. First, obviously, you can scale vertically - increasing the size of a ProxySQL instance might be the easiest way to let it handle higher traffic. Please keep in mind that ProxySQL, by default, is configured to use 4 threads.

If you want to be able to utilize more CPU cores, this is the setting you need to change as well.

Alternatively, you can attempt to scale out horizontally. Instead of using two ProxySQL instances with VIP you can collocate ProxySQL with Moodle hosts. Then you want to reconfigure Moodle to connect to ProxySQL on the local host, ideally through the Unix socket - it is the most efficient way of connecting to ProxySQL. There is not much of a configuration that we use with ProxySQL therefore using multiple instances of ProxySQL should not add too much of the overhead. If you want, you can always setup ProxySQL Cluster to help you to keep the ProxySQL instances in sync regarding the configuration.

MariaDB Cluster is Overloaded

Now we are talking about a more serious issue. Of course, increasing the size of the instances will help, as usual. On the other hand, horizontal scale out is somewhat limited because of the “safe reads” limitation. Sure, you can add more nodes to the cluster but you can use them only for the safe reads. To what extent this lets you scale out, it depends on the workload. For pure read-only workload (browsing through the contents, forums etc) it looks quite nice:

MySQL [(none)]> SELECT hostgroup, srv_host, srv_port, status, queries FROM stats_mysql_connection_pool WHERE hostgroup IN (20, 10) AND status='ONLINE';

+-----------+---------------+----------+--------+---------+

| hostgroup | srv_host      | srv_port | status | Queries |

+-----------+---------------+----------+--------+---------+

| 20        | 192.168.1.204 | 3306     | ONLINE | 5683    |

| 20        | 192.168.1.205 | 3306     | ONLINE | 5543    |

| 10        | 192.168.1.206 | 3306     | ONLINE | 553     |

+-----------+---------------+----------+--------+---------+

3 rows in set (0.002 sec)

This is pretty much a ratio of 1:20 - for one query that hits the writer we have 20 “safe reads” that can be spread across the remaining nodes. On the other hand, when we start to modify the data, the ratio quickly changes.

MySQL [(none)]> SELECT hostgroup, srv_host, srv_port, status, queries FROM stats_mysql_connection_pool WHERE hostgroup IN (20, 10) AND status='ONLINE';

+-----------+---------------+----------+--------+---------+

| hostgroup | srv_host      | srv_port | status | Queries |

+-----------+---------------+----------+--------+---------+

| 20        | 192.168.1.204 | 3306     | ONLINE | 3117    |

| 20        | 192.168.1.205 | 3306     | ONLINE | 3010    |

| 10        | 192.168.1.206 | 3306     | ONLINE | 6807    |

+-----------+---------------+----------+--------+---------+

3 rows in set (0.003 sec)

This is an output after issuing several grades, creating forum topics and adding some course content. As you can see, with such a safe/unsafe queries ratio the writer will be saturated earlier than the readers therefore scaling out by adding more nodes is not suitable.

What can be done about it? There is a setting called “latency”. As per the configuration file, it determines when it is safe to read the table after the write. When write happens, the table is marked as modified and for the “latency” time all SELECTs will be sent to the writer node. Once the time longer than “latency” passed, SELECTs from that table may again be sent to read nodes. Please keep in mind that with MariaDB Cluster, time required for writeset to be applied across all of the nodes is typically very low, counted in milliseconds. This would allow us to set the latency quite low in the Moodle configuration file, for example the value like 0.1s (100 milliseconds) should be quite ok. Of course, should you run into any problems, you can always increase this value even further.

Another option to test would be to rely purely on MariaDB Cluster to tell when the read is safe and when it is not. There is a wsrep_sync_wait variable that can be configured to force causality checks on several access patterns (reads, updates, inserts, deletes, replaces and SHOW commands). For our purpose it would be enough to ensure that reads are executed with the causality enforced thus we shall set this variable to ‘1’.

We are going to make this change on all of the MariaDB Cluster nodes. We will also need to reconfigure ProxySQL for read/write split based on the query rules, not just the users, as we had previously. We will also remove the ‘moodle_safereads’ user as it is not needed anymore in this setup.

We set up three query rules that distribute the traffic based on the query. SELECT … FOR UPDATE is sent to the writer node, all SELECT queries are sent to readers and everything else (INSERT, DELETE, REPLACE, UPDATE, BEGIN, COMMIT and so on) is sent to the writer node as well.

This allows us to ensure that all the reads can be spread across the reader nodes thus allowing horizontal scale out through adding more nodes to the MariaDB Cluster.

We hope with those couple of tips you will be able to scale out your Moodle database backend much easier and to a greater extent

Percona XtraDB Cluster based on MySQL 8.0 has been out there for some time, so it’s not surprising more and more people are attempting to migrate into it. You may wonder how to approach this process - in this short blog post we would like to give you some suggestions on how to handle the upgrade of Galera Cluster based on MySQL 5.7 to Galera Cluster based on MySQL 8.0.

First things first, you may have noticed we are talking about specific versions of Percona XtraDB Cluster - 5.7 into 8.0. This is because Galera, on which PXC is based, is an addition to MySQL and as such it has to follow all the guidelines for MySQL upgrade. Those guidelines are clear - the only supported upgrade to MySQL 8.0 is from MySQL 5.7. As the first step you have to ensure that there are no roadblocks on the MySQL side. You can refer to our previous blog post, in which we shared some tips related to upgrading MySQL from 5.7 to 8.0. Make yourself familiar with the differences between those versions, verify that neither of the incompatible changes really impact your application. Once this is confirmed, ideally through tests, we can start planning the upgrade of PXC. There are two major ways of performing the online upgrade for Galera Cluster, let’s take a look at them now.

In-Place Upgrade

While not recommended, with some it is possible to have both PXC 5.7 and PXC 8.0 nodes running in the same cluster, this allows us to perform an in-place upgrade in which we will upgrade nodes in the cluster one by one, resulting in a cluster running on PXC 8.0. There are a couple of gotchas you have to keep in mind while doing so. For starters, as we said, you have to tick all the boxes on the MySQL 8.0 upgrade checklist. Then there are a couple of steps you have to take.

First, download the percona-release package:

wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb

and install it:

dpkg -i percona-release_latest.generic_all.deb

Then, enable Percona XtraDB Cluster 8.0 in the repositories: 

percona-release enable pxc-80 release

Finally, update apt and install percona-xtradb-cluster package:

apt update

apt install percona-xtradb-cluster

While installing, you will be presented with two dialog boxes.

In the first one you can decide if you want to enable caching_sha2_password or you want to use the old mechanism for hashing passwords. For the compatibility sake we decided to use the old one for now - this is something you can always change at the later time.

The next dialog asks if we want to use existing my.cnf or replace it. Please keep in mind that in PXC 8.0 the structure of the configuration files in /etc/ directory has changed. We would probably want to keep the old my.cnf as it contains working configuration for our cluster, some tuned settings etc, but eventually this would have to be reviewed and migrated into new configuration file structure.

After PXC 8.0 is installed, before you start it, there are a couple of settings you want to check in your my.cnf. Those are changes from standard my.cnf generated by ClusterControl, your mileage may vary therefore look out for the error log in case you won’t be able to start PXC 8.0.

For starters, this will be a mixed cluster for a while therefore we have to disable cluster traffic encryption added in PXC 8.0:

pxc-encrypt-cluster-traffic=OFF

Then we had to disable following settings that were incompatible with PXC (or MySQL) 8.0:

#log_warnings=2

#ignore-db-dir=lost+found

#ignore-db-dir=.s9s_do_not_delete

#query_cache_type = 0

#query_cache_size = 0

#wsrep_convert_LOCK_to_trx=0

#wsrep_drupal_282555_workaround=0

#wsrep_causal_reads=0

#wsrep_sst_auth=user:pass

Please also make sure that you have set:

read_only=OFF

super_read_only=OFF

This may not be the case if the cluster you are upgrading has been created through the “Create Slave Cluster” job in ClusterControl.

That’s pretty much all - start PXC:

systemctl start mysql

It should finish just fine. In case you run into any sorts of troubles, try to force SST - remove the data directory on the node that you upgrade and start the mysqld process again. Please keep in mind that SST from PXC 5.7 to PXC 8.0 works just fine.

The Slave Cluster Approach

With more free nodes available, you can approach the upgrade differently: two clusters on different major versions running in parallel and connected through asynchronous replication. The main advantage of this approach is that it will let you test the PXC 8.0 more thoroughly as you will have it up and running for a while (basically for as long as you need it) and you can test your application on this cluster. At any point in time you can move some of the read-only workload to the PXC 8.0 seeing how queries perform, looking for possible errors or performance problems. If you use ClusterControl, this can be accomplished by using the “Create Slave Cluster” job.

You will be asked to decide where the initial data should come from, master PXC node or from the backup.

You will also need to pass the connectivity details like SSH user and key path.

Then you will be asked to pick the vendor and version - you probably want to use PXC 5.7 for now, you will upgrade the nodes later, testing the upgrade process itself.

Finally, you have to pass the node hostnames or IP addresses for ClusterControl to connect to and start setting the nodes up.

As the result you will have a second Percona XtraDB Cluster running in version 5.7, replicating from the original cluster. That cluster can be upgraded to the new version, as we discussed in the “In-place upgrade” section and, eventually, you can switch your traffic towards it and deprecate the old cluster.

In case of any problems with the replication link, you should be able to recreate it easily with CHANGE MASTER TO … command, executed using the credentials used for the initial setup (if this has been done from ClusterControl, you’ll see them in /etc/cmon.d/cmon_X.cnf as repl_user and repl_password, where X is the cluster ID of the master cluster).

We hope this short blog post will help you to go through the process of upgrading Percona XtraDB Cluster from version 5.7 to version 8.0.

If you have been around in the container world, you would know that it is pretty challenging to adopt a full Kubernetes automation for a clustered database system, which commonly adds a level of complexity to the container-based architecture for these stateful applications. That's where a Kubernetes Operator can help us to address this problem. A Kubernetes Operator is a special type of controller introduced to simplify complex deployments which basically extends the Kubernetes API with custom resources. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate the entire life cycle of the software it manages.

Percona XtraDB Cluster Operator is a neat way to automate the specific tasks of Percona XtraDB Cluster like deployment, scaling, backups and upgrades within Kubernetes, built and maintained by Percona. It deploys the cluster in a StatefulSet with a Persistent Volume, which allows us to maintain a consistent identity for each Pod in the cluster and our data to be maintained. 

In this blog post, we are going to test out the deployment of Percona XtraDB Cluster 8.0 in a containerized environment, orchestrated by Percona XtraDB Cluster Kubernetes Operator on Google Cloud Platform. 

Creating a Kubernetes Cluster on Google Cloud

In this walkthrough, we are going to use the Kubernetes cluster on Google Cloud because it is relatively simple and easy to get Kubernetes up and running. Login to your Google Cloud Platform dashboard -> Compute -> Kubernetes Engine -> Create Cluster, and you will be presented with the following dialog:

Just enter the Kubernetes Cluster name, pick your preferred zone and click "CREATE" (at the bottom of the page). In 5 minutes, a 3-node Kubernetes cluster will be ready. Now, on your workstation, install the gcloud SDK as shown in this guide and then pull the Kubernetes configuration into your workstation:

$ gcloud container clusters get-credentials my-k8s-cluster --zone asia-northeast1-a --project s9s-qa
Fetching cluster endpoint and auth data.
kubeconfig entry generated for my-k8s-cluster.

You should be able to connect to the Kubernetes cluster at this point. Run the following command to verify:

$ kubectl get nodes
NAME                                            STATUS   ROLES    AGE    VERSION
gke-my-k8s-cluster-default-pool-b80902cd-gp09   Ready    <none>   139m   v1.16.13-gke.401
gke-my-k8s-cluster-default-pool-b80902cd-jdc3   Ready    <none>   139m   v1.16.13-gke.401
gke-my-k8s-cluster-default-pool-b80902cd-rdv8   Ready    <none>   139m   v1.16.13-gke.401

The above output means that we are able to connect to the Kubernetes master and retrieve the Kubernetes cluster nodes. Now, we are ready to run the Kubernetes workloads. 

Deploying a Percona XtraDB Cluster on Kubernetes

For workload deployment, we are going to follow the instructions as stated in the Percona XtraDB Cluster Operator documentation. Basically, we run the following command on our workstation to create the custom resources, namespace, role-based access control and also the Kubernetes operator itself:

$ git clone -b v1.6.0 https://github.com/percona/percona-xtradb-cluster-operator
$ cd percona-xtradb-cluster-operator/
$ kubectl apply -f deploy/crd.yaml
$ kubectl create namespace pxc
$ kubectl config set-context $(kubectl config current-context) --namespace=pxc
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)
$ kubectl apply -f deploy/rbac.yaml
$ kubectl apply -f deploy/operator.yaml

Next, we have to prepare our passwords (it's called Secrets in Kubernetes term) by updating the values inside deploy/secrets.yaml in a base64 encoded format. You can use online tools like https://www.base64encode.org/ to create one or using a command-line tool like the following:

$ echo -n 'mypassword' | base64
bXlwYXNzd29yZA==

Then, update the deploy/secrets.yaml, as shown below:

apiVersion: v1
kind: Secret
metadata:
  name: my-cluster-secrets
type: Opaque
data:
  root: bXlwYXNzd29yZA==
  xtrabackup: bXlwYXNzd29yZA==
  monitor: bXlwYXNzd29yZA==
  clustercheck: bXlwYXNzd29yZA==
  proxyadmin: bXlwYXNzd29yZA==
  pmmserver: bXlwYXNzd29yZA==
  operator: bXlwYXNzd29yZA==

The above is a super simplification of secret management, where we set all passwords to be the same for all users. In production, please use a more complex password and specify a different password for every user. 

Now, we can push the secret configuration to Kubernetes:

$ kubectl apply -f deploy/secrets.yaml

Before we move forward to deploy a Percona XtraDB Cluster, we need to revisit the default deployment definition inside deploy/cr.yaml for the cluster. There are a lot of Kubernetes objects that are defined here but most of them are commented out. For our workload, we would make the modification as below:

$ cat deploy/cr.yaml
apiVersion: pxc.percona.com/v1-6-0
kind: PerconaXtraDBCluster
metadata:
  name: cluster1
  finalizers:
    - delete-pxc-pods-in-order
spec:
  crVersion: 1.6.0
  secretsName: my-cluster-secrets
  vaultSecretName: keyring-secret-vault
  sslSecretName: my-cluster-ssl
  sslInternalSecretName: my-cluster-ssl-internal
  allowUnsafeConfigurations: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: recommended
    schedule: "0 4 * * *"
  pxc:
    size: 3
    image: percona/percona-xtradb-cluster:8.0.20-11.1
    configuration: |
      [client]
      default-character-set=utf8

      [mysql]
      default-character-set=utf8

      [mysqld]
      collation-server = utf8_unicode_ci
      character-set-server = utf8
      default_authentication_plugin = mysql_native_password
    resources:
      requests:
        memory: 1G
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 6Gi
    gracePeriod: 600
  haproxy:
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.6.0-haproxy
    resources:
      requests:
        memory: 1G
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    gracePeriod: 30
  backup:
    image: percona/percona-xtradb-cluster-operator:1.6.0-pxc8.0-backup
    storages:
      fs-pvc:
        type: filesystem
        volume:
          persistentVolumeClaim:
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 6Gi
    schedule:
      - name: "daily-backup"
        schedule: "0 0 * * *"
        keep: 5
        storageName: fs-pvc

We have made some modifications to the provided cr.yaml to make it work with our application, as shown above. First of all, we have to comment out (or remove) all CPU related lines, for example [*].resources.requests.cpu: 600m, to make sure Kubernetes is able to schedule the pod creation correctly on nodes with limited CPU. Then we need to add some compatibility options for Percona XtraDB Cluster 8.0 which is based on MySQL 8.0, to work smoothly with our WordPress application that we are going to deploy later on, as shown in the following excerpt:

   configuration: |
      [client]
      default-character-set=utf8

      [mysql]
      default-character-set=utf8

      [mysqld]
      collation-server = utf8_unicode_ci
      character-set-server = utf8
      default_authentication_plugin = mysql_native_password

The above will match the MySQL server's default character set with the MySQLi PHP driver in our WordPress container. The next section is the HAProxy deployment where it is set to "enabled: true". There is also a ProxySQL section with "enabled: false" - commonly one would pick either of the reverse proxies for every cluster. The last section is the backup configuration, where we would like to have a daily backup scheduled at 12:00 AM every day and keep that last 5 backups.

We can now start to deploy our 3-node Percona XtraDB Cluster:

$ kubectl apply -f deploy/cr.yaml

The creation process will take some time. The operator will deploy the Percona XtraDB Cluster pods as a Stateful Set, which means one pod creation at a time and each Pod in the StatefulSet will be assigned an integer ordinal, from 0 up through N-1, that is unique over the set. The process is over when both operator and the Pods have reached their Running status:

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cluster1-haproxy-0                                 2/2     Running   0          71m
cluster1-haproxy-1                                 2/2     Running   0          70m
cluster1-haproxy-2                                 2/2     Running   0          70m
cluster1-pxc-0                                     1/1     Running   0          71m
cluster1-pxc-1                                     1/1     Running   0          70m
cluster1-pxc-2                                     1/1     Running   0          69m
percona-xtradb-cluster-operator-79d786dcfb-6clld   1/1     Running   0          121m

Since this operator is a custom resource, we can manipulate the perconaxtradbcluster resource to like the standard Kubernetes resource:

$ kubectl get perconaxtradbcluster
NAME       ENDPOINT               STATUS   PXC   PROXYSQL   HAPROXY   AGE
cluster1   cluster1-haproxy.pxc   ready    3                3         27h

You can also use the shorter resource name, "pxc", and try with the following commands:

$ kubectl describe pxc
$ kubectl edit pxc

When looking at the workload set, we can tell that the operator has created two StatefulSets:

$ kubectl get statefulsets -o wide
NAME               READY   AGE   CONTAINERS          IMAGES
cluster1-haproxy   3/3     26h   haproxy,pxc-monit   percona/percona-xtradb-cluster-operator:1.6.0-haproxy,percona/percona-xtradb-cluster-operator:1.6.0-haproxy
cluster1-pxc       3/3     26h   pxc                 percona/percona-xtradb-cluster:8.0.20-11.2

The operator will also create the corresponding services that will load-balanced connections to the respective pods:

$ kubectl get service
NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
cluster1-haproxy            ClusterIP      10.40.9.177    <none>          3306/TCP,3309/TCP,33062/TCP   3h27m
cluster1-haproxy-replicas   ClusterIP      10.40.0.236    <none>          3306/TCP                      3h27m
cluster1-pxc                ClusterIP      None           <none>          3306/TCP,33062/TCP            3h27m
cluster1-pxc-unready        ClusterIP      None           <none>          3306/TCP,33062/TCP            3h27m

The above output shows that the operator has created 4 services:

• cluster1-haproxy - The service for a load-balanced MySQL single-master (3306), Proxy protocol (3309) and MySQL Admin (33062) - A new administrative port introduced in MySQL 8.0.14 and later. This is the service name or cluster IP address that the applications need to connect to have a single-master connection to the Galera cluster.
• cluster1-haproxy-replicas - The service for a load-balanced MySQL multi-master (3306). This is the service name or cluster IP address that the applications need to connect to have a multi-master connection to the Galera cluster with round-robin balancing algorithm.
• cluster1-pxc - The service for load-balanced PXC pods, bypassing HAProxy. By connecting directly to this service, Kubernetes will route the connection in round-robin fashion to all PXC pods, similar to what cluster-haproxy-replicase provides. The service has no public IP address assigned and is unavailable outside the cluster.
• cluster1-pxc-unready - The 'unready' service is needed for pod address discovery during the application startup regardless of the Pod state. Proxysql and pxc pods should know about each other before the database becomes fully operational. The unready service has no public IP address assigned and is unavailable outside the cluster.

To connect via a MySQL client, simply run the following command:

$ kubectl run -i --rm --tty percona-client --image=percona:8.0 --restart=Never -- bash -il

This will create a transient Pod and immediately enter the container environment. Then, run the standard mysql client command with a proper credential:

bash-4.2$ mysql -uroot -pmypassword -h cluster1-haproxy -P3306 -e 'SELECT @@hostname'
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| @@hostname     |
+----------------+
| cluster1-pxc-0 |
+----------------+

When we look at the Pod placement, all Percona XtraDB Cluster pods are located on a different Kubernetes host:

$ kubectl get pods -o wide --selector=app.kubernetes.io/component=pxc
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                                            NOMINATED NODE   READINESS GATES
cluster1-pxc-0   1/1     Running   0          67m   10.36.2.5    gke-my-k8s-cluster-default-pool-b80902cd-gp09   <none>           <none>
cluster1-pxc-1   1/1     Running   0          66m   10.36.1.10   gke-my-k8s-cluster-default-pool-b80902cd-rdv8   <none>           <none>
cluster1-pxc-2   1/1     Running   0          65m   10.36.0.11   gke-my-k8s-cluster-default-pool-b80902cd-jdc3   <none>           <none>

This will definitely improve the availability of the service, in case one of the Kubernetes hosts goes down. 

To scale up to 5 pods, we need to prepare another 2 new Kubernetes nodes beforehand to respect the pod affinity configuration (default to affinity.antiAffinityTopologyKey.topologyKey="kubernetes.io/hostname"). Then, run the following patch command to scale the Percona XtraDB Cluster to 5 nodes:

$ kubectl patch pxc cluster1 \
--type='json' -p='[{"op": "replace", "path": "/spec/pxc/size", "value": 5 }]'

Monitor the pod's creation by using kubectl get pods command:

$ kubectl get pods -o wide
NAME               READY   STATUS      RESTARTS   AGE   IP           NODE                                            NOMINATED NODE   READINESS GATES
cluster1-pxc-0     1/1     Running     0          27h   10.36.2.5    gke-my-k8s-cluster-default-pool-b80902cd-gp09   <none>           <none>
cluster1-pxc-1     1/1     Running     0          27h   10.36.1.10   gke-my-k8s-cluster-default-pool-b80902cd-rdv8   <none>           <none>
cluster1-pxc-2     1/1     Running     0          27h   10.36.0.11   gke-my-k8s-cluster-default-pool-b80902cd-jdc3   <none>           <none>
cluster1-pxc-3     1/1     Running     0          30m   10.36.7.2    gke-my-k8s-cluster-pool-1-ab14a45e-h1pf         <none>           <none>
cluster1-pxc-4     1/1     Running     0          13m   10.36.5.3    gke-my-k8s-cluster-pool-1-ab14a45e-01qn         <none>           <none>

Another 2 new Pods (cluster1-pxc-3 and cluster1-pxc-4) have been created on another 2 new Kubernetes nodes (gke-my-k8s-cluster-pool-1-ab14a45e-h1pf and gke-my-k8s-cluster-pool-1-ab14a45e-01qn). To scale down, simply change the value back to 3 in the above patch command. Note that Percona XtraDB Cluster should be running with an odd number of nodes to prevent split-brain.

Deploying an Application (WordPress)

In this example, we are going to deploy a WordPress application on top of our Percona XtraDB Cluster and HAProxy. Let's first prepare the YAML definition file like the following:

$ cat wordpress-deployment.yaml
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: cluster1-haproxy
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-cluster-secrets
              key: root
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

Pay attention to theWORDPRESS_DB_HOST and WORDPRESS_DB_PASSWORD environment variables. The former variable where we defined "cluster1-haproxy" as the database host, instead of an individual database node and for the latter we specified the root password by instructing Kubernetes to read it from my-cluster-secrets object under key "root", which is equivalent to "mypassword" (after the base64 value was decoded). We skip defining the WORDPRESS_DB_USER environment variable since the default value is "root".

Now we can create our application:

$ kubectl apply -f wordpress-deployment.yaml

Check the service:

$ kubectl get service
NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
cluster1-haproxy            ClusterIP      10.40.9.177    <none>          3306/TCP,3309/TCP,33062/TCP   4h42m
cluster1-haproxy-replicas   ClusterIP      10.40.0.236    <none>          3306/TCP                      4h42m
cluster1-pxc                ClusterIP      None           <none>          3306/TCP,33062/TCP            4h42m
cluster1-pxc-unready        ClusterIP      None           <none>          3306/TCP,33062/TCP            4h42m
wordpress                   LoadBalancer   10.40.13.205   35.200.78.195   80:32087/TCP                  4h39m

At this point, we can connect to our WordPress application at http://35.200.78.195/ (the external IP address) and start configuring the WordPress application. At this point, our WordPress application is connected to one of the Percona XtraDB Cluster (single-master connection) via one of the HAProxy pods.

That's it for now. For more information, check out the Percona Kubernetes Operator for Percona XtraDB Cluster documentation. Happy containerizing!

Percona XtraDB Cluster is a cluster solution based on the synchronous replication handled by Galera library. It is one of the most common high availability solutions for MySQL. It supports multiple writers and the conflict resolution, making failovers a breeze - no need to check the node’s position, promote a master, reslave remaining slaves. With PXC, just like with other Galera-based solutions, you just connect to another node and continue sending traffic. All of it makes it a very popular solution in MySQL world. Let’s take a look at how you can deploy Percona XtraDB Cluster.

Before we start talking about the deployment, let’s focus first on the requirements regarding high availability. The way the clustering works implies that you should have an odd number of nodes and a three-node cluster is a minimum that allows you to automatically handle a failure of a single node. We will not go into the details here, we have blogs that cover the topic of high availability in Galera in detail, but we should assume three nodes as a minimal deployment.

How to Deploy Percona XtraDB Cluster?

Manual Deployment

There are several ways in which we can deploy a PXC. For starters, manual deployment - you want to start with a single node, bootstrap the cluster off that node and then add additional nodes to the cluster through proper configuration. This is a perfectly valid deployment method yet it requires some knowledge of the process and, the most important bit, time to perform it. Percona, of course, has everything covered in the documentation, but when you look at it, you will find that it is not the easiest process, especially when you are not familiar with Percona XtraDB Cluster or Galera in general. Sure, all steps are nicely explained but there is still a room for human error and all different troubles and, most importantly, delays may be a result of it.

Using Docker

If you use a Docker in your environment, you may consider using PXC as Docker containers. Sure, databases as all stateful services are not that easy to use within containers but it becomes more and more popular. Percona provides us with Docker images and instructions on how to use them - all can be found in the documentation. Another alternative is to use Percona Kubernetes Operator for PXC - if you use Kubernetes to orchestrate your Docker environment, utilizing the operator will help you to manage the cluster. One of my colleagues has written a blog that explains how you can use Percona Kubernetes Operator for PXC in a Google Cloud kubernetes environment. Using the operator reduces the burden of managing everything by hand as well as introducing a solution that you can depend on instead of configuring the whole environment by hand (even if in Docker).

Using Infrastructure Orchestration Tools

As you see, this leaves us with two main ways of setting up PXC - automated for Docker and manual for other types of deployments. Of course, one may argue that there are ansible playbooks or chef cookbooks available out there but, let’s be honest, this, most likely, won’t work from the get-go. First of all, you’ll waste a lot of time to find something that resembles a complete solution and then, most likely, you will have to introduce changes to customize the playbook to your liking and to your environment. This alone requires knowledge not only of PXC or Galera but also of the infrastructure orchestration tool that’s going to be used to deploy PXC. For some people it is a viable option, for others, not so much.

ClusterControl

Here comes ClusterControl. It allows you to deploy, among others, Percona XtraDB Cluster. ClusterControl does not discriminate - all environments have been created equal and as long as CC can log into them using SSH and install software using package manager, it will work just fine with bare metal, virtual machines or containers. The deployment requires some input from the user but after all data is collected, the deployment itself is fully automated. Let’s take a look at how it works.

At the beginning, as mentioned above, you have to define SSH connectivity. It has to be a passwordless key-based SSH using either root user or a sudo user (with or without sudo password).

As the next step you have to pick the vendor and version, define password for PXC root user and pick the hosts to install the cluster on. As of now, ClusterControl supports PXC 5.6 and PXC 5.7 with PXC 8.0 coming in the next release of ClusterControl (1.8.1). As you can see, the nodes are tested for SSH connectivity and if any issue shows up, it will be mentioned at this step. Once you are done with completing this form, you can click “Deploy” and enjoy your time as ClusterControl will set up the cluster for you.

If you want, you can track the progress of your deployment, following the steps ClusterControl is performing to set up your Percona XtraDB Cluster.

When everything is completed, you shall see the new cluster on the cluster list in ClusterControl, making it available for managing from within the ClusterControl platform.

If you prefer to use command line rather than the UI or if you want to integrate ClusterControl into your existing deployment scripts, you can deploy PXC using ClusterControl CLI. The CLI equivalent of the deployment that we initiated via UI would be to run following command:

s9s cluster \

--create \

--cluster-type=galera \

--nodes="10.0.0.161,10.0.0.162,10.0.0.163" \

--vendor=percona \

--provider-version=5.7 \

--db-admin=root \

--db-admin-passwd=pass \

--os-user=root \

--os-key-file=/root/.ssh/id_rsa \

--verbose \

--wait

Conclusion

As you can see, ClusterControl provides you with multiple ways to deploy Percona XtraDB Cluster, we hope you’ll find it an useful tool to spin up new PXC instances in an efficient and easy way. You can check it for yourself by downloading ClusterControl.

Percona XtraDB Cluster is a very well known high availability solution in MySQL world. It is based on Galera Cluster and it does provide virtually synchronous replication across multiple nodes. As with every database, it is crucial to keep track of what is happening in the system, if performance is on the expected levels and, if not, what is the bottleneck. This is of utmost importance to be able to react properly in the situation, where performance is impacted. Of course, Percona XtraDB Cluster comes with multiple metrics and it is not always clear which of them are the most important ones to track the state of the database. In this blog we will discuss a couple of the key metrics you want to keep an eye on while working with PXC.

To make it clear, we will be focusing on the metrics unique to PXC and Galera, we will not be covering metrics for MySQL or InnoDB. Those metrics have been discussed in our previous blogs.

Let’s take a look at some of the most important information that PXC presents to us.

Flow Control

Flow control is pretty much the most important metric you can monitor in any Galera Cluster therefore let’s have a bit of background. Galera is a multi-master, virtually synchronous cluster. It is possible to execute writes on any of the database nodes that form it. Every write has to be sent to all of the nodes in the cluster to ensure that it can be applied - this process is called the certification. No transaction can be applied before all nodes agree it can be committed. If any of the nodes has performance problems that makes it unable to cope with the traffic, it will start issuing flow control messages which are intended to inform the rest of the cluster about the performance problems and ask them to reduce the workload and help the delayed node to catch up with the rest of the cluster.

You can track when nodes had to introduce artificial pause to let their lagging peers to catch up using flow control paused metric (wsrep_flow_control_paused):

You can also track if the node is sending or receiving the flow control messages (wsrep_flow_control_recv and wsrep_flow_control_sent).

This information will help you understand better which node is not performing on the same level as its peers. You can then focus on that node and try to understand what is the issue and how to remove the bottleneck.

Send and Receive Queues

Those metrics are sort of related to the flow control. As we have discussed, a node may be lagging behind other nodes in the cluster. It can be caused by a non-even workload split or by other reasons (some process running in the background, backup or some custom, heavy queries). Before the flow control kicks in, lagging nodes will attempt to store the incoming writesets in the receive queue (wsrep_local_recv_queue) hoping that the performance impact is transient and it will be able to catch up very soon. Only if the queue becomes too big (it is governed by gcs.fc_limit setting), flow control messages start to be sent across the cluster.

You can think of a receive queue as the early marker which shows that there are problems with the performance and the flow control may kick in.

On the other hand, send queue (wsrep_local_send_queue) will tell you that the node is not able to send the writesets to other members of the cluster which may indicate problems with the network connectivity (pushing the writesets to the network is not really resource-intensive).

Parallelization Metrics

Percona XtraDB cluster can be configured to use multiple threads to apply the incoming writesets - it allows it to better handle multiple threads connecting to the cluster and issuing writes at the same time. There are two main metrics that you may want to keep an eye on.

First, wsrep_cert_deps_distance, tells us what is the parallelization potential - how many writesets can, potentially, be applied at the same time. Based on this value you can configure the number of parallel slave threads (wsrep_slave_threads) that will work on applying incoming writesets. The rule of thumb is that there is no point in configuring more threads than the value of wsrep_cert_deps_distance.

Second metric, on the other hand, tells us how efficiently we were able to parallelize the process of applying writesets - wsrep_apply_oooe tells us how often applier started to apply writesets out of order (which points towards better parallelization).

Conclusion

As you can see, there are a couple of metrics worth looking at in Percona XtraDB Cluster. Of course, as we stated at the beginning of this blog, those are metrics strictly related to PXC and Galera Cluster in general.

You should also keep an eye on regular MySQL and InnoDB metrics to get a better understanding of the state of your database. And remember, you can monitor this technology for free using the ClusterControl Community Edition.

We’re excited to announce the 1.8.1 release of ClusterControl - the only database management system you’ll ever need to take control of your open source database infrastructure. 

In addition to several new security enhancements, ClusterControl now supports the deployment, monitoring and management of Percona XtraDB Cluster 8 and MariaDB Cluster 10.5; both of which are built on the new Galera Cluster 4 technology which offers streaming replication and new synchronization functions.

This release also features the ability to cluster “fully-aware” ProxySQL nodes offering significant scalability of the popular database load balancing technology.

We are also rolling out a brand new backup and restore methodology for MongoDB which offers several advantages when backing up and restoring replica sets and sharded clusters.

Release Highlights

ProxySQL Clustering

• ProxySQL Clustering lets you have multiple ProxySQL instances which are natively clustered and aware of one other.

MongoDB Backup Management Improvements

• Added Percona Backup for MongoDB which allows you to backup and restore Replica sets and Sharded Clusters

Security Improvements

• Improved prevention for Clickjacking
• Updated JQuery to version 3.5.0 which features several new security fixes
• Implemented CGI Generic Cross-Site Request Forgery Detection. 

New Database Version Support

• Support for Percona XtraDB Cluster 8.0
• Support for MariaDB Cluster 10.5

View Release Details and Resources

• Download ClusterControl
• ChangeLog
• ClusterControl Administration Guide
• Upgrade Instructions

Release Details

ProxySQL Clustering

In April 2020, ProxySQL released version 1.4.2 which introduced the ability to natively cluster ProxySQL nodes. This feature has now been added to ClusterControl. It allows multiple ProxySQL instances to be aware of one other and let you create users and query rules that are instantly propagated to all the other nodes in the cluster.

MongoDB Backup Management

With the implementation of Percona Backup for MongoDB ClusterControl now has the ability to backup and restore both non-sharded replica sets and sharded clusters. 

Security Improvements

Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated. We have implemented the ability to detect these types of attacks improving database security. 

Clickjacking is the malicious practice of manipulating a website user's activity by concealing hyperlinks beneath legitimate clickable content, thereby causing the user to perform actions of which they are unaware. We have added new protections from this type of attack in this 

JQuery implemented a security fix in version 3.5.0 after an issue was reported that demonstrated the regex could introduce a cross-site scripting (XSS) vulnerability. ClusterControl has been improved to take advantage of this new version.

New Database Support

Galera Cluster 4, which is the basis of the technology behind both Percona XtraDB Cluster 8 and MariaDB Cluster 10.5, includes several game-changing features as part of its release. The addition of Streaming Replication allows the database to better handle large transactions by fragmenting the transitions and operations and replicating them to the slave nodes while the process is still going on.  This prevents crashes when a transaction or large query becomes too large for the normal process to complete. The release also contains a series of new synchronization functions as well as the addition of new system tables to improve the database monitoring ability. 

ClusterControl now allows you to deploy this new technology with it’s point-and-click interface, reducing the complexity of deploying and configuring the, traditional challenging, database stack.

ProxySQL is a well known load balancer in MySQL world - it comes with a great set of features that allow you to take control over your traffic and shape it however you see it fit. It can be deployed in many different ways - dedicated nodes, collocated with application hosts, silo approach - all depends on the exact environment and business requirements. The common challenge is that you, most of the cases, want your ProxySQL nodes to contain the same configuration. If you scale out your cluster and add a new server to ProxySQL, you want that server to be visible on all ProxySQL instances, not just on the active one. This leads to the question - how to make sure you keep the configuration in sync across all ProxySQL nodes?

You can try to update all nodes by hand, which is definitely not efficient. You can also use some sort of infrastructure orchestration tools like Ansible or Chef to keep the configuration across the nodes in a known state, making the modifications not on ProxySQL directly but through the tool you use to organize your environment.

If you happen to use ClusterControl, it comes with a set of features that allow you to synchronize the configuration between ProxySQL instances but this solution has its cons - it is a manual action, you have to remember to execute it after a configuration change. If you forget to do that, you may be up to a nasty surprise if, for example, keepalived will move Virtual IP to the non-updated ProxySQL instance.

None of those methods is simple or 100% reliable and the situation is when the ProxySQL nodes have different configurations and might be potentially dangerous.

Luckily, ProxySQL comes with a solution for this problem - ProxySQL Cluster. The idea is fairly simple - you can define a list of ProxySQL instances that will talk to each other and inform others about the version of the configuration that each of them contains. Configuration is versioned therefore any modification of any setting on any node will result in the configuration version being increased - this triggers the configuration synchronization and the new version of the configuration is distributed and applied across all nodes that form the ProxySQL cluster.

The recent version of ClusterControl allows you to set up ProxySQL clusters effortlessly. When deploying ProxySQL you should tick the “Use Native Clustering” option for all of the nodes you want to be part of the cluster.

Once you do that, you are pretty much done - the rest happens under the hood.

MySQL [(none)]> select * from proxysql_servers;

+------------+------+--------+----------------+

| hostname   | port | weight | comment        |

+------------+------+--------+----------------+

| 10.0.0.131 | 6032 | 0      | proxysql_group |

| 10.0.0.132 | 6032 | 0      | proxysql_group |

+------------+------+--------+----------------+

2 rows in set (0.001 sec)

On both of the servers the proxysql_servers table was set properly with the hostnames of the nodes that form the cluster. We can also verify that the configuration changes are properly propagated across the cluster:

We have increased the Max Connections setting on one of the ProxySQL nodes (10.0.0.131) and we can verify that the other node (10.0.0.132) will see the same configuration:

In case of a need to debug the process, we can always look to the ProxySQL log (typically located in /var/lib/proxysql/proxysql.log) where we will see information like this:

2020-11-26 13:40:47 [INFO] Cluster: detected a new checksum for mysql_servers from peer 10.0.0.131:6032, version 11, epoch 1606398059, checksum 0x441378E48BB01C61 . Not syncing yet ...

2020-11-26 13:40:49 [INFO] Cluster: detected a peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060, diff_check 3. Own version: 9, epoch: 1606398022. Proceeding with remote sync

2020-11-26 13:40:50 [INFO] Cluster: detected a peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060, diff_check 4. Own version: 9, epoch: 1606398022. Proceeding with remote sync

2020-11-26 13:40:50 [INFO] Cluster: detected peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060

2020-11-26 13:40:50 [INFO] Cluster: Fetching MySQL Servers from peer 10.0.0.131:6032 started. Expected checksum 0x441378E48BB01C61

2020-11-26 13:40:50 [INFO] Cluster: Fetching MySQL Servers from peer 10.0.0.131:6032 completed

2020-11-26 13:40:50 [INFO] Cluster: Fetching checksum for MySQL Servers from peer 10.0.0.131:6032 before proceessing

2020-11-26 13:40:50 [INFO] Cluster: Fetching checksum for MySQL Servers from peer 10.0.0.131:6032 successful. Checksum: 0x441378E48BB01C61

2020-11-26 13:40:50 [INFO] Cluster: Writing mysql_servers table

2020-11-26 13:40:50 [INFO] Cluster: Writing mysql_replication_hostgroups table

2020-11-26 13:40:50 [INFO] Cluster: Loading to runtime MySQL Servers from peer 10.0.0.131:6032

This is the log from 10.0.0.132 where we can clearly see that a configuration change for table mysql_servers was detected on 10.0.0.131 and then it was synced and applied on 10.0.0.132, making it in sync with the other node in the cluster.

As you can see, clustering ProxySQL is an easy yet efficient way to ensure its configuration stays in sync and helps significantly to use larger ProxySQL deployments. Let us know down in the comments what your experience with ProxySQL clustering is.

Percona XtraDB Cluster 8.0 is based on Percona Server for MySQL 8.0 embedded with Galera writeset replication API and Galera replication library, to form a highly available multi-master replication for MySQL-based database server. Percona XtraDB Cluster comes with XtraDB storage engine (a drop-in replacement of InnoDB) and follows the upstream Oracle MySQL releases very closely (including all the bug fixes in it), with some additional variables and status specific for this build.

In this blog post, we are going to look into how to deploy a Percona XtraDB Cluster 8.0 for high availability using the manual and automated way.

Notable Features

Before we move further with the deployment steps, it's worth mentioning notable features and significant improvements in this major version which is based on Percona Server for MySQL 8.0 (without Group Replication plugin) and Galera 4:

• Streaming replication - This allows writeset replication to process large data transactions in smaller, more manageable fragments, minimizing data conflicts.
• The synchronization MySQL functions for action coordination (wsrep_last_seen_gtid, wsrep_last_written_gtid, wsrep_sync_wait_upto_gtid). 
• New Galera system tables, which will help to monitor the state of the cluster under the "mysql" schema - wsrep_cluster, wsrep_cluster_members and wsrep_streaming_log.
• Gcache encryption.
• The wsrep infrastructure of Galera 4 is more robust than that of Galera 3. It features a faster execution of code with better state handling, improved predictability, and error handling.

For a complete list of changes, check out the release notes.

Manual Deployment of Percona XtraDB Cluster 8.0

Installation Steps

For the Percona XtraDB Cluster, we need at least three nodes:

• db1 (192.168.20.61)
• db2 (192.168.20.62)
• db3 (192.168.20.63)

The steps described in this section should be performed on all database nodes (db1, db2 and db3) running on CentOS 8. First, disable SELinux (otherwise IST process would fail):

$ setenforce 0
$ sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux

Install the Percona repository package:

$ yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm

The latest stable version at this point is Percona XtraDB Cluster 8.0, but by default, the repository package is configured up until version 5.7. The percona-release package contains a script that can enable additional repositories for the newer products. Let's run that script and enable the pxc80 specific repository:

$ percona-release setup pxc80
* Disabling all Percona Repositories

On RedHat 8 systems it is needed to disable dnf mysql module to install Percona-Server
Do you want to disable it? [y/N]

Choose "y" to disable the mysql DNF module. This is necessary to prioritize the Percona repository for MySQL-related packages.

Then, install the latest Percona XtraDB Cluster and Percona Xtrabackup:

$ dnf -y install percona-xtradb-cluster percona-xtrabackup-80

At this moment in time, you should get a Percona XtraDB Cluster 8.0.0-11.3 installed. All dependency packages will be installed like shared-compat, shared and client packages. We can then start the MySQL service for initial configuration:

$ systemctl start mysql

A new root password will be generated during the first startup. We need to retrieve the root password information first from the MySQL error log (default is /var/log/mysqld.log in RHEL-based systems):

$ cat /var/log/mysqld.log | grep 'temporary password'
2020-11-27T06:53:05.929661Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: <eqo+jXNw2lk

As you can see the generated password is "<eqo+jXNw2lk". Next, we need to perform a post-installation task to secure the MySQL deployment. Run the following command and answer the questions accordingly:

$ mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root:

The existing password for the user account root has expired. Please set a new password.

New password:

Re-enter new password:

VALIDATE PASSWORD COMPONENT can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD component?

Press y|Y for Yes, any other key for No: y

There are three levels of password validation policy:
LOW    Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary file

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 2

Using existing password for root.
Estimated strength of the password: 100

Change the password for root ? ((Press y|Y for Yes, any other key for No) : y

New password:

Re-enter new password:

Estimated strength of the password: 100

Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y

By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.

Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.

By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.

Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y

Success.

All done!

The generated root password will be expired immediately upon the first root login. The above helper script helps us to configure a new MySQL root password, setting password validation policy, disabling remote login for root, remove test database and anonymous users and also reload the privilege tables.

The default configuration file (/etc/my.cnf) is configured with an empty gcomm:// connection string under wsrep_cluster_address variable, as shown below:

wsrep_cluster_address=gcomm://

The above line must be set with a list of IP addresses of all participating nodes in the cluster. If you missed this step, a new Galera cluster would be created on every node (3 separate 1-node cluster) after the service on every nodes was started. Therefore, the following lines should be modified accordingly under [mysqld] section:

wsrep_cluster_address=gcomm://192.168.20.51,192.168.20.52,192.168.20.53
wsrep_node_address=192.168.20.51 # IP address of this DB host
wsrep_node_name=db1 # any name to identify this DB host
pxc-encrypt-cluster-traffic=OFF

Do expect for the wsrep_cluster_address to be identical in all database hosts, while wsrep_node_address and wsrep_node_name must be distinct on every database host. The last line pxc-encrypt-cluster-traffic=OFF means we would like to disable encryption for the cluster traffic. Note that this is intentional to simplify our deployment steps, or otherwise we would have to perform key/cert management steps. If you would like to enable encryption, see this article.

Repeat the above steps for the remaining database nodes before proceeding to bootstrap the cluster.

Cluster Bootstrapping

Galera requires you to start a node in a cluster as a reference point before the remaining nodes are able to join and form the cluster. This process is known as cluster bootstrap. Bootstrapping is an initial step to introduce a database node as the primary component before others see it as a reference point to sync up data.

In this example, we are going to use db1 as the reference node and start it with the bootstrap command (you could choose other nodes as well since this is a new cluster with no data):

$ systemctl stop mysql
$ systemctl start mysql@bootstrap
$ systemctl status mysql@bootstrap

Once started, we can start the other nodes (db2 and db3) with the standard start command (one node at a time):

$ systemctl stop mysql
$ systemctl start mysql

You can verify with the following line inside /var/log/mysqld.log:

2020-11-27T08:09:01.964152Z 2 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections

At this point, our Percona XtraDB Cluster service has started and the manual deployment is now complete. For more information regarding Percona XtraDB Cluster bootstrapping process, check out this blog post How to Bootstrap MySQL or MariaDB Galera Cluster. 

Automatic Deployment with ClusterControl

There are two ways you can deploy a Percona XtraDB Cluster 8.0 using ClusterControl. You may use the ClusterControl UI (web-based GUI) or ClusterControl CLI called s9s. We will show you both ways in this section. ClusterControl must reside on a separate host, away from your database cluster. Therefore, our architecture can be illustrated like this:

Installing ClusterControl

SSH into the ClusterControl server and install ClusterControl with the following commands:

$ wget http://severalnines.com/downloads/cmon/install-cc
$ chmod 744 install-cc
$ ./install-cc

Follow the installation wizard until the installation finishes. After installation is complete, open the web browser and go to http://192.168.20.19/clustercontrol . Create an admin user and the license will be installed automatically. All new installations come with a trial license for enterprise edition for 30 days. Once this trial license expires, it will default back to the community edition which is free forever.

Passwordless SSH

ClusterControl requires passwordless SSH to be set up from the ClusterControl server to all the managed nodes. To achieve this, we need to generate an SSH key on the ClusterControl server first. The SSH user must have a super-user privilege (sudo or root). In this case, we are going to use the root user:

$ whoami
root

Now generate an SSH key for user root:

$ ssh-keygen -t rsa # press Enter on all prompts

Next, copy the public key to all the database nodes that we are going to deploy and manage. In this case, all 3 Galera nodes:

$ ssh-copy-id root@192.168.20.51
$ ssh-copy-id root@192.168.20.52
$ ssh-copy-id root@192.168.20.53

Enter the root password of the respective hosts when prompted.

Test it out with the following command and make sure you get a correct response:

$ ssh root@192.168.10.101 "ls /root"

Now we are ready to deploy our cluster using ClusterControl UI or CLI, as shown in the next sections.

PXC Deployment using ClusterControl UI

It's practically easy to deploy a Percona XtraDB Cluster with ClusterControl, and by default, ClusterControl will configure the cluster with Galera encryption enabled. After the passwordless SSH has been configured, go to ClusterControl -> Deploy -> MySQL Galera and specify the required details as below:

Then, click "Continue" to proceed to the next step where we configure the MySQL specification:

Choose "Percona" for the Vendor and 8.0 as the Version. Keep the rest as default and enter the MySQL root password. Specify the IP address or hostname of the database hosts one by one and make sure you get the green tick icons after each insertion. This indicates that ClusterControl is able to reach the corresponding hosts via passwordless SSH with the provided SSH user and key as defined in step 1. Click on the "Deploy" button to start the deployment.

ClusterControl then triggers a deployment job where you can monitor the deployment progress by going to ClusterControl -> Activity -> Jobs -> Create Cluster -> Full Job Details, as shown in the following screenshot:

Once the process completes, you should see the cluster is listed in the Dashboard:

That's it. The deployment is now complete.

PXC Deployment using ClusterControl CLI

If you prefer to deploy the cluster using command-line, we can simply use the ClusterControl command-line client tool called "s9s". This tool will send a deployment job to the ClusterControl controller and it will perform all the necessary steps to deploy the cluster.

Run the following command on the ClusterControl server:

$ s9s cluster --create \
--cluster-type=galera \
--nodes="192.168.20.51;192.168.20.52;192.168.20.53" \
--vendor=percona \
--provider-version=8.0 \
--db-admin-passwd='P4ssw0rdSecr3t' \
--os-user=root \
--os-key-file=/root/.ssh/id_rsa \
--cluster-name='My Percona XtraDB Cluster 8.0' \
--log

You will see the job messages shall appear and you can monitor the deployment progress there and you can also monitor the job progress from ClusterControl UI -> Activity -> Jobs. Wait for 15-20 minutes (depending on the Internet connection) until you see the job completion notification. Our Percona XtraDB Cluster 8.0 is now deployed and you can see it listed in the ClusterControl UI.

ClusterControl 1.8.1 includes support for MariaDB Cluster 10.5. MariaDB 10.5 is equipped with.

• More Granular Privileges
• InnoDB Performance Improvements
• Full GTID Support for Galera Cluster
• More Metadata for Replication and Binary Logs
• More SQL syntax statements are introduced (RETURNING statement to INSERT, EXCEPT ALL and INTERSECT ALL, …)
• Performance Schema Updates to Match MySQL 5.7
• The S3 Storage Engine

You can check further on our previous blog What’s New in MariaDB Server 10.5? to know more details about the release. With the specific updates for MariaDB Cluster, in version 10.5, the key features don’t differ much from MariaDB Cluster 10.4, but the list below are some of the important changes in this version.

• GTID consistency 
• Cluster inconsistency/error voting 
• non-blocking DDL operations (available only on enterprise version)
• Black box (available only on enterprise version) 
• Upgraded its Galera wsrep Library for which 26.4.6 is the latest version

XA Transaction Support is expected on this release ( do not be confused as XA Transactions are supported by MariaDB Server but not on Galera Cluster) but due to some reasons it's delayed and shall be expected on the next MariaDB Cluster 10.6 release.

MariaDB Cluster For High Availability

The MariaDB Cluster is basically a Galera Cluster that uses the MariaDB implementation as the database layer to interface with Innodb or XtraDB engine. MariaDB Galera Cluster is a virtually synchronous multi-master cluster for MariaDB. It is available on Linux only, and only supports the XtraDB/InnoDB storage engines (although there is experimental support for MyISAM - see the wsrep_replicate_myisam system variable). When Galera Cluster is in use, database reads and writes can be directed to any node. Any individual node can be lost without interruption in operations and without using complex failover procedures.

With Galera's nature adapted within MariaDB Cluster, it's basically a high availability solution with synchronous replication, failover, and resynchronization. It brings the benefits of no data loss, no slave lag, read and write scalability, and does high availability on different data centers.

Deploying MariaDB Cluster 10.5

MariaDB provides a very straightforward and easy setup for installing your MariaDB Cluster 10.5. The manual process can be tedious but with automated scripts provided by MariaDB, repositories can be setup in accordance to your target database version, OS type, and OS version.

For this exercise, I have the following 3-node Galera Cluster setup with the following IP addresses: 192.168.40.210, 192.168.40.220, 192.168.40.230.

Setup Your Repository

As mentioned earlier, MariaDB has a script named mariadb_repo_setup. It's very straightforward and easy to use. You can specify the target version of your database, the OS type, and the version of your OS. 

For example, I am installing using CentOS 8,

curl -LsS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup |  sudo bash -s -- --mariadb-server-version="mariadb-10.5" --os-type=rhel --os-version=8

or installing it in Ubuntu Focal Fossa,

curl -LsS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup |  sudo bash -s -- --mariadb-server-version="mariadb-10.5" --os-type=ubuntu --os-version=focal

Take note that when using mariadb_repo_setup script, it requires the package apt-transport-https as a dependency. So install this package first before you can take advantage of the mariadb_repo_setup script. 

apt update

apt install apt-transport-https

Now, run the command to your three nodes in accordance to their OS and of course the MariaDB version has to be 10.5.

Setup MySQL Configuration

Configuration file depends on your server resources, type of server environment, and assigned IP address. For this blog, you can use this production ready MariaDB Cluster/PXC configuration setup which we used to deploy in our Percona XtraDB Cluster/MariaDB Cluster databases using ClusterControl. Notable variables that you need or subject to change are the following:

• innodb_buffer_pool_size - set the buffer pool from 70% - 80% available RAM of your server
• wsrep_provider - Path of the Galera compiled library. For RHEL/CentOS, the path shall be  /usr/lib64/galera-4/libgalera_smm.so. Whereas Debian/Ubuntu is in /usr/lib/galera/libgalera_smm.so.
• wsrep_node_address - This is the node IP address
• wsrep_sst_method - either you can change it but we recommend you use mariabackup. Possible values you can choose are rsync, mysqldump, xtrabackup, xtrabackup-v2, mariabackup.
• wsrep_cluster_name - The name of your MariaDB Cluster. It has to be identical to all your nodes in a single MariaDB Cluster.
• wsrep_cluster_address - This contains the addresses of your nodes within the cluster. It has to be a valid IP, hostname, or FQDN.
• wsrep_node_name - The name of your node. The name can be used in wsrep_sst_donor as a preferred donor. Note that multiple nodes in a cluster can have the same name.

For performing SST, the user and password for the following sections [mysqldump], [xtrabackup], and [mysqld] can change if you want. For this exercise, let's keep it simple and you can just let the values as is.

Now, copy the configuration file and place it to /etc/my.cnf. Do this to all of your three Galera nodes.

Installing Required Packages

Install the packages for all the three Galera nodes. Follow the command below based on your target OS environment.

For RHEL/CentOS,

sudo yum install MariaDB-server MariaDB-client galera-4 MariaDB-backup

For Debian/Ubuntu,

sudo apt update

sudo apt-get install mariadb-server galera-4 mariadb-client libmariadb3 mariadb-backup mariadb-common

Once installation is done, stop the MariaDB process and initialize the cluster as a single node.This shall bootstrap your Galera Cluster. At this stage, I'm running it on node 192.168.40.210,

$ /usr/bin/galera_new_cluster

Create SST/IST User

Create the backup user which shall be used for SST or IST. Only run the following SQL statements below on the first node that you have initiated the cluster. At this stage, I have executed it in node 192.168.40.210.

CREATE USER backupuser@localhost IDENTIFIED BY 'backuppassword';

GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'backupuser'@'localhost';

At this point, we're already setup and ready to connect the remaining nodes.

Start The MariaDB Server

Now we have already setup the first node. We are ready to connect the remaining nodes. Simply, just start the mariadb service. Just run the command below,

systemctl start mariadb

Run the command for the remaining nodes. Do it one at a time.

At this point, all nodes are in synced and we can verify with the following SQL statement as follows,

Adding Steroids For Your MariaDB Cluster 10.5 for HA

Likely, in production setup, deploying the MariaDB Cluster 10.5 for HA might not suffice your needs. Adding more steroids such as installing HAProxy together with Keepalived for your redundancy, will bring more high availability for your database environment.

Setting up your HAProxy and then Keepalived adds more hassle to obtain the desired topology and environment. This can be achieved with automation using ClusterControl. Using ClusterControl allows you to deploy MariaDB Cluster 10.5 and add more solutions such as ProxySQL, MaxScale, or garbd for load balancing. While you can add Keepalived to your cluster to add redundancy and auto failover solutions in case of disaster happens.

You can download ClusterControl, which is free and allows you to deploy these HA solutions for free and manage your database giving you a 30-day subscription license to enjoy the benefits of advanced and enterprise features.

Deploying MariaDB 10.5 Cluster with ClusterControl

Once you have installed ClusterControl, click the icon from the upper right corner and you'll see the deployment Wizard just like below.

You can setup easily. Just follow the series of steps based on the flow of the UI,

Deploy HAProxy For Load Balancing Management

At this point, I would assume you have your MariaDB Cluster 10.5 all setup. Now, lets deploy the HAProxy,

Alternatively, you can go to Manage → Load Balancer → HAProxy.

Then select or type the address where the HAProxy to be installed and select your Galera nodes that have to be monitored by the HAProxy. See example below,

Add at least two deployments of HAProxy to add more availability such that whenever one of your HAProxy goes down, your application will route over to the other node that is still available or online. This is very important especially when you are handling database or system upgrades, aside from catastrophic or disaster events.

Deploy Keepalived

The same process you can use with how you deploy HAProxy with Clustercontrol. Below is a perfect example on you are going to deploy Keepalived,

If you noticed, I have two HAProxy instances for which I am going to install my Keepalived which shall be present in every node where the HAProxy is running.

Finalizing your MariaDB Cluster 10.5 With High Availability

Now that we have already setup, you have environment which shall look like this,

Conclusion

This setup gives you the benefits of achieving high availability with more 9's so to speak. HAProxy gives your MariaDB Cluster 10.5 more load balancing capability with its read and write separation. Then, Keepalived assures that in case one of your HAProxy dies, Keepalived assures that it will fail over to the next one available. Your application shall only connect to the virtual IP address (which follows the VRRP) and no extra configuration or setup to be done in case of such HAProxy instance failure occurs.

Of course, this setup is not perfect when you speak of high availability. You can replace HAProxy with ProxySQL to add more flexibility and add more read/write separation by just playing only in one port. Achieving a perfect setup for HA is hard to achieve and comes with drawbacks as well. But this setup, it gives you more high presence from your application and database interoperability. What matters most is that a lower downtime to a possible no downtime has to be achieved.

The MySQL database workload is determined by the number of queries that it processes. There are several situations in which MySQL slowness can originate. The first possibility is if there are any queries that are not using proper indexing. When a query cannot make use of an index, the MySQL server has to use more resources and time to process that query. By monitoring queries, you have the ability to pinpoint SQL code that is the root cause of a slowdown and fix it before the overall performance degrades.

In this blog post, we are going to highlight the Query Outlier feature available in ClusterControl and see how it can help us improve the database performance. In general, ClusterControl performs MySQL query sampling in two ways:

1. Fetch the queries from the Performance Schema (recommended).
2. Parse the content of MySQL Slow Query.

If the Performance Schema is disabled, ClusterControl will then default to the Slow Query log. To learn more on how ClusterControl performs this, check out this blog post, How to use the ClusterControl Query Monitor for MySQL, MariaDB and Percona Server.

What are Query Outliers?

An outlier is a query that takes a longer time than the normal query time of that type. Do not literally take this as "badly written" queries. It should be treated as potential suboptimal common queries that could be improved. After a number of samples and when ClusterControl has had enough stats, it can determine if latency is higher than normal (2 sigmas + average_query_time) then it is an outlier and will be added into the Query Outlier.

This feature is dependent on the Top Queries feature. If Query Monitoring is enabled and Top Queries are captured and populated, the Query Outliers will summarize these and provide a filter based on timestamp. To see the list of queries that require attention, go to ClusterControl -> Query Monitor -> Query Outliers and should see some queries listed (if any):

As you can see from the screenshot above, the outliers are basically queries that took at least 2 times longer than the average query time. First the first entry, the average time is 34.41 ms while the outlier's query time is 140 ms (more than 2 times higher than the average time). Similarly, for the next entries, the Query Time and Avg Query Time columns are two important things to justify the outstandings of a particular outlier query. 

It is relatively easy to find a pattern of a particular query outlier by looking at a bigger time period, like a week ago, as highlighted in the following screenshot:

By clicking on each row, you can see the full query which is really helpful to pinpoint and understand the problem, as shown in the next section.

Fixing the Query Outliers

To fix the outliers, we need to understand the nature of the query, the tables' storage engine, the database version, clustering type and how impactful the query is. In some cases, the outlier query is not really degrading to the overall database performance. As in this example, we have seen that the query has been standing out for the whole week and it was the only query type being captured so it is probably a good idea to fix or improve this query if possible.

As in our case, the outlier query is:

SELECT i2l.country_code AS country_code, i2l.country_name AS country_name 
FROM ip2location i2l 
WHERE (i2l.ip_to >= INET_ATON('104.144.171.139') 
AND i2l.ip_from <= INET_ATON('104.144.171.139')) 
LIMIT 1 
OFFSET 0;

And the query result is:

+--------------+---------------+
| country_code | country_name  |
+--------------+---------------+
| US           | United States |
+--------------+---------------+

Using EXPLAIN

The query is a read-only range select query to determine the user's geographical location information (country code and country name) for an IP address on table ip2location. Using the EXPLAIN statement can help us understand the query execution plan:

mysql> EXPLAIN SELECT i2l.country_code AS country_code, i2l.country_name AS country_name 
FROM ip2location i2l 
WHERE (i2l.ip_to>=INET_ATON('104.144.171.139') 
AND i2l.ip_from<=INET_ATON('104.144.171.139')) 
LIMIT 1 OFFSET 0;
+----+-------------+-------+------------+-------+--------------------------------------+-------------+---------+------+-------+----------+------------------------------------+
| id | select_type | table | partitions | type  | possible_keys                        | key         | key_len | ref  | rows  | filtered | Extra                              |
+----+-------------+-------+------------+-------+--------------------------------------+-------------+---------+------+-------+----------+------------------------------------+
|  1 | SIMPLE      | i2l   | NULL       | range | idx_ip_from,idx_ip_to,idx_ip_from_to | idx_ip_from | 5       | NULL | 66043 |    50.00 | Using index condition; Using where |
+----+-------------+-------+------------+-------+--------------------------------------+-------------+---------+------+-------+----------+------------------------------------+

The query is executed with a range scan on the table using index idx_ip_from with 50% potential rows (filtered).

Proper Storage Engine

Looking at the table structure of ip2location:

mysql> SHOW CREATE TABLE ip2location\G
*************************** 1. row ***************************
       Table: ip2location
Create Table: CREATE TABLE `ip2location` (
  `ip_from` int(10) unsigned DEFAULT NULL,
  `ip_to` int(10) unsigned DEFAULT NULL,
  `country_code` char(2) COLLATE utf8_bin DEFAULT NULL,
  `country_name` varchar(64) COLLATE utf8_bin DEFAULT NULL,
  KEY `idx_ip_from` (`ip_from`),
  KEY `idx_ip_to` (`ip_to`),
  KEY `idx_ip_from_to` (`ip_from`,`ip_to`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

This table is based on the IP2location database and it is seldomly being updated/written, usually only on the first day of the calendar month (recommended by the vendor). So one option is to convert the table to MyISAM (MySQL) or Aria (MariaDB) storage engine with fixed row format to get better read-only performance. Note that this is only applicable if you are running on MySQL or MariaDB standalone or replication. On Galera Cluster and Group Replication, please stick to the InnoDB storage engine (unless you know what you are doing).

Anyway, to convert the table from InnoDB to MyISAM with fixed row format, simply run the following command:

ALTER TABLE ip2location ENGINE=MyISAM ROW_FORMAT=FIXED;

In our measurement, with 1000 random IP address lookup tests, the query performance improved around 20% with MyISAM and fixed row format:

• Average time (InnoDB): 21.467823 ms
• Average time (MyISAM Fixed): 17.175942 ms
• Improvement: 19.992157565301 %

You can expect this result to be immediate after the table is altered. No modification on the higher tier (application/load balancer) is necessary.

Tuning the Query

Another way is to inspect the query plan and use a more efficient approach for a better query execution plan. The same query can also be written using subquery as below:

SELECT `country_code`, `country_name` FROM 
  (SELECT `country_code`, `country_name`, `ip_from` 
   FROM `ip2location` 
   WHERE ip_to >= INET_ATON('104.144.171.139') 
   LIMIT 1) 
AS temptable 
WHERE ip_from <= INET_ATON('104.144.171.139');

The tuned query has the following query execution plan:

mysql> EXPLAIN SELECT `country_code`,`country_name` FROM 
(SELECT `country_code`, `country_name`, `ip_from` 
FROM `ip2location` 
WHERE ip_to >= INET_ATON('104.144.171.139') 
LIMIT 1) 
AS temptable 
WHERE ip_from <= INET_ATON('104.144.171.139');
+----+-------------+--------------+------------+--------+---------------+-----------+---------+------+-------+----------+-----------------------+
| id | select_type | table        | partitions | type   | possible_keys | key       | key_len | ref  | rows  | filtered | Extra                 |
+----+-------------+--------------+------------+--------+---------------+-----------+---------+------+-------+----------+-----------------------+
|  1 | PRIMARY     | <derived2>   | NULL       | system | NULL          | NULL      | NULL    | NULL |     1 |   100.00 | NULL                  |
|  2 | DERIVED     | ip2location  | NULL       | range  | idx_ip_to     | idx_ip_to | 5       | NULL | 66380 |   100.00 | Using index condition |
+----+-------------+--------------+------------+--------+---------------+-----------+---------+------+-------+----------+-----------------------+

Using subquery, we can optimize the query by using a derived table that focuses on one index. The query should return only 1 record where the ip_to value is greater than or equal to the IP address value. This allows the potential rows (filtered) to reach 100% which is the most efficient. Then, check that the ip_from is less than or equal to the IP address value. If it is, then we should find the record. Otherwise, the IP address does not exist in the ip2location table.

In our measurement, the query performance improved around 99% using a subquery:

• Average time (InnoDB + range scan): 22.87112 ms
• Average time (InnoDB + subquery): 0.14744 ms
• Improvement: 99.355344207017 %

With the above optimization, we can see a sub-millisecond query execution time of this type of query, which is a massive improvement considering the previous average time is 22 ms. However, we need to make some modifications to the higher tier (application/load balancer) in order to benefit from this tuned query.

Patching or Query Rewriting

Patch your applications to use the tuned query or rewrite the outlier query before it reaches the database server. We can achieve this by using a MySQL load balancer like ProxySQL (query rules) or MariaDB MaxScale (statement rewriting filter), or using the MySQL Query Rewriter plugin. In the following example, we use ProxySQL in front of our database cluster and we can simply create a rule to rewrite the slower query into the faster one, for example:

Save the query rule and monitor the Query Outliers page in ClusterControl. This fix will obviously remove the outlier queries from the list after the query rule is activated.

Conclusion

Query outliers is a proactive query monitoring tool that can help us understand and fix the performance problem before it is getting way out of control. As your application grows and becomes more demanding, this tool can help you maintain a decent database performance along the way.

If you are dealing with a bunch of virtual machines, using Vagrant is probably the best tool to build, automate and manage the virtual machines to increase efficiency and productivity, especially in a development environment. ClusterControl can also be configured to run on Vagrant and can be automated easily using the Vagrantfile together with an automation script. 

Vagrant supports a number of virtualization platforms; e.g., VirtualBox, KVM, Hyper-V, Docker containers, VMware and AWS. In this example, we are going to use VirtualBox as the virtual machine provider and use Vagrant to provision the virtual machine host automatically.

Download VirtualBox from here, and download Vagrant from here. Install them into your workstation and you are good to go.

Deploying ClusterControl on Vagrant

To deploy a ClusterControl virtual machine, simply create the following files:

• Vagrantfile - Virtual machine definition and specification
• deploy-cc.sh - A bash script to install ClusterControl

Create a directory to store the Vagrant definition file:

$ mkdir ~/vagrant/clustercontrol
$ cd ~/vagrant/clustercontrol

Create the Vagrantfile definition file:

$ vim Vagrantfile

And add the following lines:

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  config.vm.network "forwarded_port", guest: 80, host: 8080
  config.vm.network "forwarded_port", guest: 443, host: 8443
  config.vm.network :private_network, ip: "192.168.11.11"
  config.vm.hostname = "cc.local"
  config.vm.provision "shell" do |s|
    s.inline = "yum install -y vim wget"
  end
  config.vm.provision "shell", path: "deploy-cc.sh"

  config.vm.provider :virtualbox do |vb|
    vb.customize [
      "modifyvm", :id,
      "--memory", "4096",
    ]
  end
end

Create the deployment script, deploy-cc.sh in the same directory. This script will be executed automatically every time the virtual machine starts up:

$ vim deploy-cc.sh

And add the following lines:

CC_IP=192.168.11.11 # IP address of the ClusterControl VM
CMON_PASS='cmonP455&' # CMON user password
ROOT_PASS='S3cr3t&' # MySQL root password
MYSQL_PORT=3306

# only install ClusterControl if cmon does not exist
if [ ! -e /usr/sbin/cmon ]; then

    # download installer script
    sudo wget severalnines.com/downloads/cmon/install-cc
    # make it executable
    sudo chmod 755 install-cc
    # execute the installation
    sudo S9S_CMON_PASSWORD=${CMON_PASS} S9S_ROOT_PASSWORD=${ROOT_PASS} S9S_DB_PORT=$MYSQL_PORT HOST=${CC_IP} ./install-cc

    # generate an SSH key for user vagrant
    ssh-keygen -t rsa -N '' -C vagrant@cc -f /home/vagrant/.ssh/id_rsa
    # assign correct ownership to the private and public keys
    sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa
    sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa.pub

    # optional - copy the public key to Apache docroot so others can download it for authorization
    cat /home/vagrant/.ssh/id_rsa.pub | sudo tee /var/www/html/cc.pub
    # optional - assign correct ownership to the public key
    sudo chown -f apache:apache /var/www/html/cc.pub
fi

We are now ready to start our first instance. We can check the VM status by using the following command:

$ vagrant status

To create and start the VM, run the following command:

$ vagrant up

You should see a lot of lines coming up on the screen indicating the VM is being prepared by Vagrant. Wait until the process completes, and you should be able to access the ClusterControl UI at http://127.0.0.1:8080/ or https://127.0.0.1:8443/ of your local machine. Port 8080 is for plain HTTP connection while 8443 is for HTTPS, as defined inside the Vagrantfile under "forwarded_port" lines.

Deploying Database Clusters on Vagrant

Now we have the ClusterControl provisioned and ready to manage and monitor our development database cluster environment. Creating multiple virtual machines using Vagrant is easy using the Vagrantfile. In this example, we would like to deploy a 3-node Galera Cluster running on Percona XtraDB Cluster 8.0 as illustrated in the following diagram:

The cluster consists of 3 nodes of database nodes and 2 nodes of ProxySQL nodes for MySQL load balancing. This whole set can be considered as one group of nodes for this cluster. Therefore, we are going to define the VMs specification in one single Vagrantfile. Firstly, create a directory for our database cluster:

$ mkdir ~/vagrant/clustercontrol/percona-xtradb-cluster-80/
$ cd ~/vagrant/clustercontrol/percona-xtradb-cluster-80/

Then, create a Vagrantfile definition using your preferred text editor for our 3-node database cluster and 2-node ProxySQL cluster:

$ vim ~/vagrant/clustercontrol/percona-xtradb-cluster-80/Vagrantfile

And add the following lines:

nodes = [
  { :hostname => 'db1', :ip => '192.168.11.21', :ram => 1024, :guestport => 3306, :hostport => 13306 },
  { :hostname => 'db2', :ip => '192.168.11.22', :ram => 1024, :guestport => 3306, :hostport => 23306 },
  { :hostname => 'db3', :ip => '192.168.11.23', :ram => 1024, :guestport => 3306, :hostport => 33306 },
  { :hostname => 'proxy1', :ip => '192.168.11.26', :ram => 512, :guestport => 6033, :hostport => 16033 },
  { :hostname => 'proxy2', :ip => '192.168.11.27', :ram => 512, :guestport => 6033, :hostport => 26033 }
]

Vagrant.configure("2") do |config|
  nodes.each do |node|
    config.vm.define node[:hostname] do |nodeconfig|
      nodeconfig.vm.box = 'centos/7'
      nodeconfig.vm.hostname = node[:hostname] + ".local"
      nodeconfig.vm.network :private_network, ip: node[:ip]
      nodeconfig.vm.network "forwarded_port", guest: node[:guestport], host: node[:hostport]

      memory = node[:ram] ? node[:ram] : 512;
      nodeconfig.vm.provider :virtualbox do |vb|
        vb.customize [ "modifyvm", :id, "--memory", memory.to_s ]
        vb.customize [ "modifyvm", :id, "--audio", "none" ]
      end

      nodeconfig.vm.provision "shell" do |s|
        s.inline = "yum install -y wget vim"
      end
    end
  end

  (1..3).each do |i|
    config.vm.define "db#{i}" do |dbnode|
      dbnode.vm.provision "shell", path: "authorize.sh"
    end
  end

  (1..2).each do |i|
    config.vm.define "proxy#{i}" do |dbnode|
      dbnode.vm.provision "shell", path: "authorize.sh"
    end
  end

end

In the same directory, create a shell script called authorize.sh and add the following lines:

CC_IP=192.168.11.11
curl -s http://${CC_IP}/cc.pub >> /home/vagrant/.ssh/authorized_keys

This script will pull the public key from the ClusterControl server (as configured inside deploy-cc.sh when we provision our ClusterControl VM), and store it inside the authorized key list for the OS user "vagrant". This is basically to automatically configure the passwordless SSH which is mandatory for the nodes to be managed by ClusterControl.

To create and start the VMs for our cluster, run the following command:

$ vagrant up

Wait for a moment until all VMs are up and running. You can verify with the vagrant status command:

$ vagrant status

Once done, it's time to deploy our Percona XtraDB Cluster using ClusterControl. Open ClusterControl at http://127.0.0.1:8080/clustercontrol and go to Deploy -> MySQL Galeraand enter the following details:

We specified vagrant as the SSH user because we have auto-authorized the public key in every nodes' authorized_keys list when provisioning the VMs using Vagrant using the authorize.sh script. Note that every Vagrant address VM for Virtualbox will have at least two network interfaces. With VirtualBox, Vagrant requires the first network device attached to the virtual machine to be a NAT device. The NAT device is used for port forwarding, which is how Vagrant gets SSH access to the virtual machine. Our IP address as defined in the Vagrantfile will be added as in interface eth1, eth2 and so on, which is crucial when defining them under the "Define MySQL Servers" section as shown below:

When specifying the IP address in the second step above, use the IP address of what we have defined inside Vagrantfile (which is equivalent to eth1's IP address of the created VM). The green tick next to the IP address indicates the node is reachable via passwordless SSH from the ClusterControl node. Finally, click "Deploy" and wait until the deployment completes.

You should see the following once the deployment job is complete:

Deploying Load Balancers on Vagrant

Similarly, for load balancers, we have already prepared the virtual machine using Vagrant with the proper IP address and passwordless SSH. To deploy a ProxySQL, go to ClusterControl -> Manage -> Load Balancers -> ProxySQL -> Deploy ProxySQL and specify the first load balancer, 192.168.11.26 as below:

Do not forget to toggle on the "Use Native Clustering" since we are going to have more than one ProxySQL host running. Click on the "Deploy ProxySQL" button to start the load balancer deployment. You can monitor the job progress under ClusterControl -> Activity -> Jobs. Repeat the above steps for the second load balancer, 192.168.11.27. Finally, we can tie these load balancers together with a virtual IP address (a floating IP address for a single endpoint) using Keepalived by going toClusterControl -> Manage -> Load Balancers -> Keepalived -> Deploy Keepalived and choose both of the load balancers with a virtual IP address (192.168.11.20) on the interface eth1, as shown below:

Our database cluster deployment with a high-availability setup similar to our production cluster is now complete.

To shut down all the database nodes, we can just simply run the following command in the active directory:

$ cd ~/vagrant/clustercontrol/percona-xtradb-cluster-80
$ vagrant halt

To shut down the ClusterControl node, navigate back to the ClusterControl directory and run the same command:

$ cd ~/vagrant/clustercontrol
$ vagrant halt

Substitute the "vagrant halt" command with the "vagrant destroy" command if you want to destroy the virtual machines (usually for housekeeping purposes) and you can always re-create it back later using the "vagrant up" command.

Open edX is a platform for online educational activities. Given the situation the world is in, all such platforms are encountering higher loads and their importance has significantly increased. Those are not just the “helper” platforms but they often become the main way in which educational activities are performed. This leads to the higher requirements regarding the load they can handle or the availability of the platform.

Open edX is a complex product that consists of multiple elements. One of them is the MySQL database. In this short blog we would like to discuss how you can improve the high availability of the Open edX platform.

Obviously, the single MySQL database is a single point of failure and, as such, it is not an approach suitable for the production deployments. There are several ways in which you can improve the availability of MySQL database.

For starters, you can run the master - slave setup using asynchronous or semi-synchronous replication. The advantage of it is that, when the master is unavailable, you can promote one of the slaves and proceed with the operation. The main downside of such a setup is that the failover has to be performed either manually, which increases the downtime or you have to use some external software (for example ClusterControl) but then again it still may take a bit of time. Finally, any kind of asynchronous or semi-synchronous replication is subject to replication lag. This may seriously impact the read-after-write scenarios in which the application executes a write on the master and then immediately attempts to read that data from the slave.

Another approach would be to use a Galera Cluster to store the data from Open edX platform. We can start with three node clusters - such clusters can automatically handle the failure of a single node. The remaining two nodes will continue to work and be responsive for queries coming from the application. Another advantage of Galera is that, even though it is “virtually” synchronous (which means pretty much that it is almost synchronous), there is a way to enforce causality checks and force clusters into the synchronous mode (even if you pay for it with reduced performance).

Both scenarios would require some sort of a load balancer in front of them, which would handle the traffic and redirect it to a proper destination.

Let’s see how ClusterControl can help you to deploy a Galera Cluster with a set of load balancers that you can use for your Open edX platform. 

Deploying MariaDB Cluster

This time we’ll try to use MariaDB Cluster as our backend. Again, the first step is the same, we need to pick “Deploy” from the wizard:

Once you do that, we have to define SSH connectivity, passwordless, key-based SSH access is a requirement for ClusterControl, otherwise it won’t be able to manage database infrastructure.

Then we should decide on the vendor, version, password,  hosts and some additional settings:

With all those details filled we are good to proceed with deployment.

Deploying ProxySQL

Database itself is not the only element that we want to deploy. We also need a load balancer which we will use to direct the traffic to the nodes that are available in the given moment. We will also use it to provide read/write split, directing all of the writes to a single MariaDB Galera node. This will help us avoid conflicts between writes executed on different Galera nodes.

For ProxySQL ClusterControl also requires filling in some information - you have to pick the host to install it on, decide on ProxySQL version, credentials for the administrative and monitoring users. Those users will be used to manage ProxySQL and monitor the state of your Galera cluster. You should also import existing database users or create a new one for your application. Finally, it is up to you to decide which database nodes you want to use with ProxySQL and decide if you use implicit transactions.

Deploying Keepalived

As the next step we will deploy Keepalived. The idea here is to have a virtual IP that will point to the working ProxySQL instance. Such VIP can then be used in the application as the endpoint for the MySQL database connectivity.

After passing details like ProxySQL instances that should be monitored, Virtual IP and the interface VIP should bind to we are ready to deploy. After couple of minutes everything should be ready and the topology should look like below:

That’s pretty much it when it comes to the deployment. You can point your Open edX platform towards the VIP and port 6033, this should be enough to get the connectivity to your backend database. The last remaining bit, should you find it necessary, would be to configure causality checks. There is a wsrep_sync_wait variable that can do just that. It can enable tests on several access patterns: reads, updates, inserts, deletes, replaces and SHOW commands. If we are interested only in the SELECT queries, we will set this variable to ‘1’ using ClusterControl configuration management.

This will perform this change on all of the MariaDB Cluster nodes.

That’s pretty much it. If you would like to share some of your experience with Open edX, you are welcome to leave us a comment.

There is a huge demand for a highly available Moodle system to have a Galera Cluster as the database backend due to its growing popularity and robustness. Galera Cluster turns the popular MySQL and MariaDB server into a multi-master replication with an automated member provisioning and joining process and no data loss guarantee with a proper configuration. However, in a busy Moodle environment, a multi-writer Galera Cluster could face a degrading issue due to high contention on hotspot tables, especially sessions and caches.

In this blog post, we are going to look at how to install and configure Redis to offload our Galera Cluster from session management data in Moodle. Our architecture consists of a single Moodle web server, connected to a three-node MariaDB Cluster via an HAProxy instance. The HAProxy configuration is set for multi-writer configuration, where every Galera node will process read/write in a round-robin fashion. 

Moodle's Session and Cache Management

A PHP session stores data on the server rather than the user's computer. In a session-based environment, every user is identified through a unique number called session identifier or SID. This unique session ID is used to link each user with their own information on the server. Without a PHP session, a user will be logged out of the Moodle system.

Moodle needs to store the session data in storage. It supports a number of persistent storage endpoints like filesystem, database, memcached and Redis, to store the following components:

• Default file store for application caches.
• Default session store for session caches.

We chose Redis because it is simple to install (available directly from the OS package manager) and easy to configure out-of-the-box.

Installing Redis

You can use standalone Redis or a clustered Redis called Redis Sentinal. For simplicity reasons, we are going to deploy only a single standalone Redis server on the same host as the Moodle server. The Redis session driver is available only on Moodle 3.1.3 onwards, which requires a Redis server and the Redis PHP extension to be installed.

First of all, install php-redis extension:

$ apt -y install php-redis

Load the module into Apache by restarting the Apache webserver:

$ systemctl restart apache2

Install Redis tools and server:

$ apt -y install redis

Modify the following line inside /etc/redis/redis.conf, to protect the Redis server with password authentication and configure systemd as the supervision tree:

$ vim /etc/redis/redis.conf
requirepass redispass123
supervised systemd

Restart Redis to load the new changes:

$ systemctl restart redis

Test connecting to the Redis server via redis-cli:

$ redis-cli
127.0.0.1:6379> AUTH redispass123
OK
127.0.0.1:6379> 

We are now good to configure Moodle's session and cache management to Redis. 

Configure Moodle with Redis

Log in to Moodle as administrator, then go to Site administrator -> Plugins -> Caching -> Configuration. Under "Installed cache stores" section, click on the "Add instance" for Redis and enter the following details:

The Password text field is equivalent to the requirepass value inside /etc/redis/redis.conf, which in this case is "redispass123". Click "Save Changes". Next, go to the "Stores used when no mapping is present" section (at the bottom of the page) and click on the "Edit mappings", and set the "Application" and "Session" as the following:

Click on the "Save changes" button. Our Moodle system is now integrated with the Redis server for caches and session storage. To test, log in to the Moodle system and you can then check if the Redis server has stored the session and/or cache data by using the redis-cli command line:

$ redis-cli
127.0.0.1:6379> AUTH redispass123
OK
127.0.0.1:6379> KEYS *
  1) "lciaqttsndkgoa6b61th03ad7o"
  2) "mo8bbm8hoonug4b3dndsbitpto"
...
...

Or if you prefer the GUI, you may use phpRedisAdmin to get the same result:

Our Redis installation and integration with the Moodle system is now complete. 

Performance Improvement

To get a summary of the Redis performance, we used the Redis Stats, where it will produce a summary like this:

The key components that we need to focus on are the Hits and Misses values, the Used and Peak Memory values and the number of keys stored in db0 (default for Moodle). Make sure the hit rate percentage is high around the peak (the pie chart), indicating an efficient cache system, while the memory section shows how much memory to store and serve the 202 keys in db0, giving a good visualization of the total cache size and memory capacity allocation of the server.

When comparing the database queries with and without Redis, depending on the user's activities, we could see up to 10 database hits for the same session ID on multiple tables. The following lines are taken from MySQL's general log involving all queries related to session ID 'fc02352nsgrohlhnv961p7blm1':

2021-01-21T12:12:12.985149Z      6403 Query     INSERT INTO mdl_sessions (state,sid,sessdata,userid,timemodified,timecreated,lastip,firstip) VALUES('0','fc02352nsgrohlhnv961p7blm1',NULL,'26','1611231132','1611231132','13.229.218.239','13.229.218.239')
2021-01-21T12:12:13.031471Z      6404 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:13.048408Z      6405 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:27.407469Z      6447 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:42.708191Z      6482 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:55.205215Z      6525 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:55.205977Z      6525 Query     SELECT * FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:55.206212Z      6525 Query     DELETE FROM mdl_external_tokens WHERE sid = 'fc02352nsgrohlhnv961p7blm1' AND tokentype = '1'
2021-01-21T12:12:55.206482Z      6525 Query     DELETE FROM mdl_sessions WHERE sid = 'fc02352nsgrohlhnv961p7blm1'
2021-01-21T12:12:55.221436Z      6525 Query     INSERT INTO mdl_logstore_standard_log (eventname,component,action,target,objecttable,objectid,crud,edulevel,contextid,contextlevel,contextinstanceid,userid,courseid,relateduserid,anonymous,other,timecreated,origin,ip,realuserid) VALUES ('\\core\\event\\user_loggedout','core','loggedout','user','user','26','r','0','1','10','0','26','0',NULL,'0','{\"sessionid\":\"fc02352nsgrohlhnv961p7blm1\"}','1611231175','web','13.229.218.239',NULL)

With Redis is enabled, regardless of the user's activities, it will only make 2 database queries per session ID, only on the mdl_sessions table, as shown below for session ID '00b3fsn4k78oj9dq0oeo1j39pi':

2021-01-21T12:38:44.060790Z      9381 Query     INSERT INTO mdl_sessions (state,sid,sessdata,userid,timemodified,timecreated,lastip,firstip) VALUES('0','00b3fsn4k78oj9dq0oeo1j39pi',NULL,'0','1611232724','1611232724','13.229.218.239','13.229.218.239')
2021-01-21T12:38:44.090198Z      9382 Query     SELECT id, sid, state, userid, lastip, timecreated, timemodified FROM mdl_sessions WHERE sid = '00b3fsn4k78oj9dq0oeo1j39pi'

From this observation, we can tell that with the help of Redis, we would see a steady 2 database queries per session, otherwise, the database server has to process more queries for cache and session handling, which is up to 5 times if Redis is out of the picture.

In conclusion, when configuring Moodle to use Redis for session and cache storage, we can save a notable amount of database hits, especially on the hotspot tables like mdl_sessions and mdl_external_tokens. This configuration will significantly improve the Galera Cluster replication performance by reducing the number of write conflicts for a multi-master configuration. In Galera Cluster, due to cluster-wide optimistic locking, the conflicted transaction will have to retry (default is wsrep_retry_autocommit=1) or rollback (after out of retry), depending on the database server's configuration. Note that a rollback is generally a slower operation than a commit and when rollbacks keep happening, it will have an impact on the replication performance as a whole.

Note that, for a busy Moodle system, it still can cause problems with the concurrency especially the contention on the mdl_sessions table because the current implementation still leaves some parts of the session managed by the core functions. There is an improvement request here to offload all session-related data management to Redis and we hope it will get implemented in the near future.

A hybrid infrastructure environment is a mix of active hosts residing on both on-premises and cloud infrastructures in a single operational or distributed system. Percona XtraDB Cluster, one of the Galera Cluster variants, can be configured to run on a hybrid infrastructure environment with a bit of tuning to ensure it can deal with most of the uncertainties when involving multiple sites.

In this blog post, we will look into how to deploy Percona XtraDB Cluster on a hybrid infrastructure environment, between on-premises and cloud infrastructure running on AWS. This setup allows us to bring the database closer to the clients and applications residing in a cloud environment while keeping a copy of the database on the on-premises for disaster recovery and live backup purposes.

Consideration for a Hybrid Environment

In a hybrid environment, it is better to use an easily distinguishable fully-qualified domain name (FQDN) for every database node in a cluster, instead of using the IP address as an identifier. The best way to achieve this by using the host mapping technique, where we define all the hosts inside /etc/hosts on all nodes in the cluster:

# content of /etc/hosts on every node

54.129.6.197    cc.pxc-hybrid.mycompany.com cc
18.131.73.6 db1.pxc-hybrid.mycompany.com db1

18.116.111.116 db2.pxc-hybrid.mycompany.com db2

202.86.57.37 db3.pxc-hybrid.mycompany.com db3

Make sure the firewalls are configured to allow communication between these ports - 3306, 4444, 4567, 4568. If you have ClusterControl to manage the cluster, consider allowing port 9999 and SSH (default is 22).

Note that Galera Cluster is sensitive to the network due to its virtually synchronous nature. The farther the Galera nodes are in a given cluster, the higher latency and its write capability to distribute and certify the writesets. Also, if the connectivity is not stable, cluster partitioning can easily happen, which could trigger cluster synchronization on the joiner nodes. In some cases, this can introduce instability to the cluster.

Some recommendations are to increase Galera's heartbeat interval and health check timeout parameters to tolerate for the transient network connectivity failures between geographical locations. The following parameters can tolerate 30 second connectivity outages:

wsrep_provider_options = "{any existing_configurations};evs.keepalive_period = PT3S;evs.suspect_timeout = PT30S;evs.inactive_timeout = PT1M; evs.install_timeout = PT1M"

In configuring these parameters, consider the following:

• Set the evs.keepalived_period (default is PT1S)  to at least higher than the highest round trip time (RTT) value. The default value of 1 second is good for 10-100 milliseconds latency. A simple rule of thumb is 5 to 10 times the max RTT of the ping report.

• Set the evs.suspect_timeout (default is PT5S) as high as possible to avoid partitions. Partitions cause state transfers, which can affect performance.

• Set the evs.inactive_timeout (default is PT15S) to a value higher than that of the evs.suspect_timeout  (default is PT5S).

• Set the evs.install_timeout (default is PT15S) to a value higher than the value of the evs.inactive_timeout (default is PT15S).

By increasing the above parameters, we can reduce the risk of cluster partitioning, which can help to stabilize the cluster in a non-stable network. 

Also, consider setting a different gmcast.segment value on the database node that resides on the secondary site. As in this example, we are going to set gmcast.segment=1 on db3, while the rest of the nodes will use the default value 0. With this configuration, optimizations on communication are performed to minimise the amount of traffic between network segments including writeset relaying and IST and SST donor selection.

Deploying a Hybrid Percona XtraDB Cluster 

Our deployment consists of 4 nodes - One node for ClusterControl and 3 nodes Percona XtraDB Cluster, where 2 of the database nodes are located in the AWS data center as the primary datacenter (as well as ClusterControl), while the third node is located in the on-premises datacenter. The following diagram illustrates our final architecture:

All instructions provided here are based on Ubuntu Server 18.04 machines. We will deploy our cluster using ClusterControl Community Edition, where the deployment and monitoring features are free forever.

First of all, we need to modify the MySQL configuration template that will be used for the deployment. Some amendments are required to tune for our hybrid infrastructure setup, as discussed in the previous section. 

Start by creating the custom template directory under /etc/cmon directory:

$ sudo mkdir -p /etc/cmon/templates

Then, copy the default template file under /usr/share/cmon/templates for our target deployment, Percona XtraDB Cluster 8.0 called my.cnf.80-pxc. We are going to name it as my.cnf.80-pxc-hybrid:

$ sudo cp /usr/share/cmon/templates/my.cnf.80-pxc /etc/cmon/templates/my.cnf.80-pxc-hybrid

Then modify the content using your favorite text editor:

$ sudo vim /etc/cmon/templates/my.cnf.80-pxc-hybrid

From:

wsrep_provider_options="base_port=4567; gcache.size=@GCACHE_SIZE@; gmcast.segment=@SEGMENTID@;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem @WSREP_EXTRA_OPTS@"

To:

wsrep_provider_options="evs.keepalive_period = PT3S;evs.suspect_timeout = PT30S;evs.inactive_timeout = PT1M; evs.install_timeout = PT1M;base_port=4567; gcache.size=@GCACHE_SIZE@; gmcast.segment=@SEGMENTID@;socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem @WSREP_EXTRA_OPTS@"

Now, our configuration template is ready to be used for the deployment. Next is setting up passwordless SSH from ClusterControl nodes to all database nodes:

$ whoami

ubuntu

$ ssh-keygen -t rsa # press Enter on all prompts

Retrieve the public key content on ClusterControl host:

$ cat /home/ubuntu/.ssh/id_rsa.pub

Paste the output of the above command inside /home/ubuntu/.ssh/authorized_keys on all database nodes. If the authorized_keys file does not exist, create it beforehand as the corresponding sudo user. In this example, the user is "ubuntu":

$ whoami

ubuntu

$ mkdir -p /home/ubuntu/.ssh/

$ touch /home/ubuntu/.ssh/authorized_keys

$ chmod 600 /home/ubuntu/.ssh/authorized_keys

We can verify if passwordless SSH is configured correctly by running the following command on the ClusterControl host:

$ ssh -i /home/ubuntu/.ssh/authorized_keys ubuntu@db1.pxc-hybrid.mycompany.com "sudo ls -al /root"

Make sure you see a directory listing for the root user without specifying any password. This means our passwordless configuration is correct.

Go to ClusterControl -> Deploy -> MySQL Galera and specify the SSH credentials as follows:

On the second step, "Define MySQL Servers", specify the template that we have prepared under /etc/cmon/templates/my.cnf.80-pxc-hybrid, and specify the FQDN of all database nodes, as highlighted in the following screenshot:

Make sure you get a green tick icon next to the database FQDN name, indicating ClusterControl is able to connect to the database nodes to perform the deployment. Click on the "Deploy" button to start the deployment and you can monitor the deployment progress under Activity -> Jobs -> Create Cluster. After the deployment is finish, you should see the database cluster is listed in the ClusterControl dashboard:

As the final step, we have to define a different segment ID for db3 since it is located away from the majority of the nodes. To achieve this, we need to append "gmcast.segment=1" setting to the wsrep_provider_options on db3. You can perform the modification manually inside the MySQL configuration file, or you can use ClusterControl Configuration Manager under ClusterControl -> Manage -> Configurations:

Do not forget to restart the database node to apply the changes. In ClusterControl, just to go to Nodes -> choose db3 -> Node Actions -> Restart Node -> Proceed to restart the node gracefully.

At a glance, we can tell that db3 is located in a different network segment, indicated by a higher ping RTT value:

With ClusterControl, our database cluster on a hybrid environment can be easily deployed with full visibility on the whole topology. The applications/clients can now connect to both Percona XtraDB Cluster nodes in the primary site (AWS), while the third node in the secondary site (on-premises datacenter) can also serve as the recovery site when necessary.

The term “hybrid” is popular  nowadays. Hybrid is used for vehicles, applications, financials, and also cloud. For example, in the vehicle's use case, the hybrid means combining the power of the gasoline engine with an electric motor.

In the hybrid cloud environment, we combine and connect the resources between a private cloud or on-prem environment with the public cloud. One popular use case is to mirror an on-prem environment in the cloud for disaster recovery purposes. There are some points you need to consider when building a Hybrid Cloud database. Latency will determine which MariaDB architecture you can use. A reliable connection with low and predictable latency means you can spread one Galera Cluster across both environments, with the DR setup in the cloud being syncronously up-to-date with your on-prem environment. However, this also means that the performance of the entire cluster will be limited by the performance of the slowest node in the cluster.

Another alternative is to have two separate systems that are connected using regular asynchronous replication. For instance, it is possible to have two MariaDB Galera clusters asynchronously replicate with each other. For those who prefer the standard asynchrnous replication, we propose two master-slave setups, with the second setup replication from the first one.

In this blog, we will provide a quick hands on guide on how to run a highly available MariaDB replicated setup in a Hybrid Cloud environment.

Why Hybrid Cloud?

Hybrid Cloud enables enterprise organizations to mix the environment between the private on-prem and public cloud.  This model provides the following benefits for the organization :

• Scalability of infrastructure

You can quickly scale the infrastructure by combining private cloud and public cloud as the business rapidly grows. The public cloud offers a cost effective way to extend your infrastructure, while in a private setup, it requires upfront planning and CAPEX.

• Disaster Recovery 

A hybrid cloud can be categorized as having a Disaster Recovery Plan, with regards to the deployment model. Public clouds can be used as disaster recovery sites, if something happens to the private datacenter (eg. force majeure, data center issue).

• Better technical control and security

By having a hybrid cloud environment, organizations are able to segregate environments. Share the load of services based on restricted access and also enable multi tenancy and segregating the layer.

• Architectural Flexibility

Running hybrid cloud environments gives you flexibility in how you can design services based on the workload and the requirements from the application side. For example, a private cloud environment can be restricted with regards to access to Internet, except to a public cloud environment via VPN, while the public cloud environment handles communication to the third party services.

Connectivity

Running a hybrid cloud for databases needs a secure communication link between the private cloud and public cloud. Most of the cloud providers have some sort of connectivity option available, for instance AWS has AWS Direct Connect.

Achieving Hybrid Cloud using ClusterControl

There are a few deployment models for MariaDB in hybrid cloud environments. We can use MariaDB Master/Slave replication or MariaDB Galera Cluster. The difference between Master/Slave and Galera Cluster is the synchronization method. Master/Slave replication uses asynchronous replication of data that is written to a binlog, while MariaDB Galera Cluster uses “virtually” synchronous replication by broadcasting writesets to all nodes. It is also possible to have separate Galera Clusters replicate asynchronously via standard replication.

Deployment of MariaDB Master/Slave Replication on hybrid cloud in ClusterControl is straightforward. You just go through Deploy menu as shown below:

After clicking Deploy, choose MySQL Replication and fill the SSH user, password, and Cluster Name as shown below:

Then click Continue. Choose MariaDB as the database vendor and version to be installed. There are custom options for data directory and server port, or can use the default values.

Fill in the root password for the database, and then click Continue. Add the IP Addresses of the hosts on private and public clouds as shown below:

Note that you will need to take care of the connectivity between the private and public environments, and make sure it is secure. Then click Deploy, it will deploy MariaDB Master/Slave Replication in your hybrid cloud environment. Both environments will have a replicated setup, and the DR setup in the public cloud will be replicating asynchronously from the primary setup in your private datacenter.

Technology consumers and users have embraced cloud computing ever since its boom.  It implicates a very beneficial part nowadays to the technology on how humans use and interact with technology, especially the advent of IoT (Internet of Things) and IoE (Internet of Everything). Both technologies conceptualize how it will interconnect a variety of devices (including smartphones, home appliances, sensors, and other network devices), people, data, and processes and allow them to communicate with each other seamlessly. 

Although IoT or IoE is not our focus on this blog post, both are using cloud computing to facilitate the system that underlies its concept. Now it might have to focus on the general term of cloud computing, but a hybrid cloud vs full public cloud differentiates the importance and its advantages on where it applies to. Security, cost, data autonomy, and management can be some of the primary aspects to look into what are the pros and cons of the two.

Deploying your database technology into a Hybrid Cloud environment entails advantages but also disadvantages that covers up the whole process of this setup. Determining whether you require a Hybrid Cloud requires a full understanding of the capacity and assessment of your overall organizational or company requirements. In this blog, we'll take a look at how Hybrid Cloud and Full Public Cloud databases can be advantageous or disadvantageous in real world scenarios.

What is a Hybrid Cloud?

In our previous blog, we defined Hybrid Cloud Database as the following: 

"It is a topology that uses a mix of private, and public cloud, and even on-premises services. It sounds similar to a Multi-Cloud environment, but the main difference is that this concept refers to the combinations of public and private specifically, which could include on-prem too."

Implementing your Hybrid Cloud has to be properly documented as it can take multiple unique services to deliver its needs and satisfy the requirements. Security and data management or data autonomy has to be effective and productive. To make things clearer and simpler, let's go over its pros and cons (or the advantages and disadvantages) of the hybrid cloud.

Pros With Hybrid Cloud

The most common denominators for using Hybrid Cloud are as follows:
 

• Disaster Recovery (DR): Hybrid Cloud approach allows you to have a Disaster  Recovery environment for High Availability or redundancy, so you can failover in case of failure. Data can be distributed across multiple cloud providers for highly effective redundancy, failover, and disaster recovery.

• Scalability: In case you need to scale your databases, using a Hybrid topology allows you to scale in different ways, depending on the solution and requirements. Also, it allows you to cope with variations in resource demand adding or removing resources/nodes in a fast way.

• Faster Development: Hybrid cloud can help you deliver new products and services more quickly by eliminating barriers that slow your business and development teams down.

• Compliance and security: Confidential applications can be operated privately while less sensitive workloads can be deployed to a public cloud. Also, Hybrid Cloud infrastructure can be tailored to conform to strict, mandatory privacy, and governance rules for sensitive and confidential workloads and data.

• No lock-in: If you adopt a cloud-only model, it could be hard to avoid getting locked into one or two cloud providers. The cost of moving data from the cloud provider could be high, so using a Hybrid environment can help with this.

• Cost: In general, having this kind of mixed environment decreases the cost of maintaining hardware, and also, using a Hybrid Cloud Database environment in the public cloud will decrease it even more as you will be using a pay-as-you-go model.

Whilst, on the business side, common advantages with Hybrid Cloud are,

• Reduction in CapEx (Capital Expenditures) or the organizational IT infrastructure costs

• Improving the quality and availability of service.

• Avoiding vendor lock-in.

• Quicker actions to adapt to the changing demands in the market, with better flexibility and agility.

• The fast and quick building of innovative enhancements or advancements especially in the advanced analytical services such that involves AI that might be difficult to implement in existing environments.

• Improving transparency regarding costs and resource consumption.

• Compliance regarding data security and data sovereignty with global or regional regulations.

In design and development, common drivers are:

• Lessen the development cycle time spent, while aiming to convert classic and old approach with automation and accelerating application rollouts to achieve faster time to market

• Take advantage of existing API's, avoid reinventing from scratch. This saves cost, development time, and effort

• Accelerating the provisioning of compute and storage resources

While the most important thing, especially with a production environment, has to have a full introspection of your environment. Who accesses it (clients, applications), how it can be accessed, who's allowed (engineers) to access it, level of security, and how data is stored securely.

• Ensuring consistent authentication, authorization, auditing, and policies across computing environments.

• Using consistent tooling and processes to limit complexity.

• Providing visibility across environments.

Cons With Hybrid Cloud

Hybrid Cloud solutions especially with your database servers hosted or running in it is not a walk in the park. As there is no perfect solution for your database, , following are the denominators to show the cons with hybrid cloud:

• Make sure it works for the company: You need to make sure that it is a good solution for your company, otherwise, it could be a problem that could affect the systems or even the budget in a bad way.

• Security:  You need to ensure that your data is safe, so security is a must in all companies. Using a Hybrid Environment makes this task much more complex. Some industries, such as finance and healthcare, require very high levels of security and are subject to strict privacy controls. So, in some cases, using a Public Cloud is just not possible. Although there are things such that services that are a factor to your internal services hosted privately, services running in the Public Cloud can be used as long as it does not affect security concerns.

• Scalability limitations: In a Public Cloud Environment, most probably you can scale-up and scale-out as much as you can pay for it, but in a Private Cloud or On-Prem there is a hardware limitation and this could be a time-consuming and costly task. Aside from this, make sure that all channels that are interacting with the outside world or with your public cloud have to be in the most safest and secure way. This can also cause security breaches if not addressed properly.

• Cost: As we said, using a Public Cloud model could decrease the cost of your infrastructure, but a Private Cloud is not something cheap, so you need to check the numbers and take the best decision taking the budget into consideration.

• Infrastructure complexity: For a hybrid cloud to deliver maximum benefit, its public and private components must be strongly linked and orchestrated. Management, integration, and security become increasingly complex as the number of clouds proliferates, especially when sourced from different providers.

• Maintenance: It covers cost in this regard as you have to employ special skilled engineers to always maintain and make sure that all the infrastructure and software are running properly and accordingly to the plan. You have always made sure that nothing goes south and it always has to make sure that security measures are all tied up. Take note that most of the security breaches come from human errors or unpatched security vulnerabilities.

Full Public Cloud

When you say a Full Public Cloud, this means that you are 100% relying on the services that a public cloud vendor is offering. Regardless of how many cloud providers you have in contract, there's always a pros and cons with this.

Most popular cloud providers are Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure. Other competitors include IBM Cloud, VMware Cloud, Oracle Cloud, Alibaba Cloud and others. Public Cloud nowadays are sprouting and it's almost a common service offered especially by platform services that support software as a service or database as a service such as MariaDB Cloud, Aiven, TimeScale Cloud, or CCX.

Pros Within Full Public Cloud

• Less Hassle Infrastructure Management: Most notable aspect of having a Full Public Cloud is the advantage of less hassle or to be almost hassle-free with managing the infrastructure. Although there are servers that you have to maintain, especially the none-fully managed services, the underlying infrastructure is not your concern. If you are using fully-managed services such as DBaaS, you only have to focus on your data and how to tune up your database server you have provisioned.

• Pay-as-you-go flexibility: Imagine, if you are in a meeting discussing architectural design, you can leverage the appliances that are available to test and try on the fly. The advantage is that you are doing that by demand and on the spot. You don't have to wait for your servers to arrive or buy the special servers digging some research on what type of servers and hardware you need. Then in a real-world case, it's easy to scale and focus on your application optimization and use what are the required services or appliances by just subscribing to what you call a pay-as-you-go model or services that would satisfy your needs. This helps reduce time to wait and speed up your development processes or flow.

• Costs Reduction: You have to pay for the services you only rendered. You don't need an underlying software or licenses to facilitate and also the hardware to run it.

• 24/7 Support and Maintenance: Public Cloud support and maintenance features are usually sophisticated and are extensive. They are tested to make sure that your appliance rendered is always ensured high availability at all times. In case of disaster, you can call the hotline or ask for some help and you can ensure that the provider will take care of your problem as a top priority.

• High scalability: Scalability wise, a Public Cloud is a perfect solution with less hassle and just hopped on the business side problems to take care. Scaling your infrastructure either vertical or horizontal can be very expensive if you have to take this either on-prem. Although a private cloud can also take care of this, with on-premise the cost and problems will just arise and extend. From hardware and engineering costs, that will piled up for sure.

• Disaster Recovery: The traditional approach to DR requires a significant investment of time and resources. Companies or organizations aim to have a DR standing by as a replica of the current primary infrastructure as possible and that comes with the whole infrastructure from hardware and software. With Public Cloud, this is easy to achieve as there are available compute storage and servers that you can avail and have sophisticated network services to offer. Backup solutions are also available that are very flexible to setup for the users. These come up with easy to build DR environments with little upfront costs through a pay-as-you-go model

• Highly Available, Fault Tolerant and Lower Latency: Such that it's good for DR, Public Cloud providers invest tons of money to provide a highly available setup by extending its DR setup by different data centers in a region, and that adds a fault tolerant environment not obvious to the consumers. In that regard, you can also spread your servers into different regions and that also allows lower network latencies. You can host by region to provide a good user experience to your customers.

Cons With Full Public Cloud

A Full Public Cloud emphasizes a lot of advantages that are very beneficial. However, it's not always a perfect solution and infrastructure to utilize. Here are some common cons with availing of a Full Public Cloud.

• Privacy and Data Security: This is the most common factor that organizations and companies are dealing with. Public Clouds do not disclose or share how they are storing the data that are said to be kept private from the world and the users. You might never know how your data is being handled or managed as you do not have control or access to limit only people who can gain access. Organizations or companies that are dealing with PCI DSS or HIPAA compliance, might find using the public cloud a challenge, since storing your data without freedom and confidentiality on how your data shall be stored, access, backup, or transmitted over the wire counts every point of security concerns.

• Lack of options: Public cloud providers usually have a one-size-fits-all approach with standard options. If a company has a unique need, they may not be able to meet those requirements.

• Physical Control:When you outsource to the public cloud, it’s literally out of reach. Any configuration and other aspects of IT management are left to a group that is not directly involved in daily operations. 

Conclusion

You might have both options to choose from but it depends on what kind of requirements you need and have to implement. Public Cloud has been improving and has also adopted security measures and regulations that could help you manage your compliance with law and regulations. However, you have also to consider the cost and data autonomy whenever you need full control since you own the data and has to kept it with top notch security and control.

Auditing is the monitoring and recording of selected user database actions. It is typically used to investigate suspicious activity or monitor and gather data about specific database activities. For example, the database administrator can gather statistics about which tables are being updated, how many operations are performed, or how many concurrent users connect at particular times.

In this blog post, we are going to cover the fundamental aspects of auditing our open-source database systems, especially MySQL, MariaDB, PostgreSQL and MongoDB. This article is targeted towards DevOps engineers who commonly have less experience or exposure in audit compliance best-practice and good data governance when managing the infrastructure primarily for the database systems.

Statement Auditing

MySQL Statement Auditing

MySQL has the general query log (or general_log), which basically records what mysqld is doing. The server writes information to this log when clients connect or disconnect, and it logs each SQL statement received from clients. The general query log can be very useful when troubleshooting but not really built for continuous auditing. It has a big performance impact and should be only enabled during short time slots. There are other options to use performance_schema.events_statements* tables or Audit Plugin instead.

PostgreSQL Statement Auditing

For PostgreSQL, you can enable the log_statment to "all". Supported values for this parameter are none (off), ddl, mod, and all (all statements). For "ddl", it logs all data definition statements, such as CREATE, ALTER, and DROP statements. For "mod", it logs all DDL statements, plus data-modifying statements such as INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. 

You probably need to configure the related paratemeters like log_directory, log_filename, logging_collector and log_rotation_age, as shown in the following example:

log_directory     = 'pg_log'
log_filename      = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_statement     = 'all'
logging_collector = on
log_rotation_age  = 10080 # 1 week in minutes 

The above changes require a PostgreSQL restart, so please plan carefully before applying to your production environment. You can then find the current logs under the pg_log directory. For PostgreSQL 12, the location is at /var/lib/pgsql/12/data/pg_log/ . Note that the log files tend to grow a lot over time, and might eat the disk space significantly. You could also use log_rotation_size instead if you have limited storage space.

MongoDB Statement Auditing

For MongoDB, there are 3 logging levels that can help us audit the statements (operations or ops in MongoDB term):

• Level 0 - This is the default profiler level where the profiler does not collect any data. The mongod always writes operations longer than the slowOpThresholdMs threshold to its log. 

• Level 1 - Collects profiling data for slow operations only. By default slow operations are those slower than 100 milliseconds. You can modify the threshold for “slow” operations with the slowOpThresholdMs runtime option or the setParameter command. 

• Level 2 - Collects profiling data for all database operations.

To log all operations, set db.setProfilingLevel(2, 1000), where it should profile all operations with operations that take longer than the milliseconds defined, in this case, is 1 second (1000 ms). The query to look in the system profile collection for all queries that took longer than one second, ordered by timestamp descending will be. To read the operations, we can use the following query:

mongodb> db.system.profile.find( { millis : { $gt:1000 } } ).sort( { ts : -1 } )

Also, there is Mongotail project, which simplifies the operation profiling process with an external tool instead of querying directly to the profile collection.

Bear in mind that it is not recommended to run full statement auditing in the production database servers because it commonly introduces a significant impact to the database service with an enormous volume of logging. The recommended way is to use a database audit plugin instead (as shown further down), which provides a standard way of producing audit logs often required to comply with government, financial, or ISO certifications.

Privilege Auditing for MySQL, MariaDB and PostgreSQL

Privilege auditing audits the privileges and access control to the database objects. Access control ensures that the users accessing the database are positively identified and can access, update, or delete the data that they are entitled to. This area is commonly being overlooked by the DevOps engineer which makes over-privileging a common mistake when creating and granting a database user. 

Examples of over-privileged are:

• User's access hosts are allowed from a very broad range, for example granting user host app1@'%', instead of an individual IP address.

• Administrative privileges being assigned to non-administrative database users, for example, a database user for application is being assigned with SUPER or RELOAD privilege. 

• Lack of resource control against any kind of excessive usage like Max User Connections, Max Queries Per Hour, or Max Connections Per Hour.

• Allow specific database users to access other schemas as well.

For MySQL, MariaDB and PostgreSQL, you can perform privilege auditing via the Information Schema by querying the grant, role and privileges-related tables. For MongoDB, use the following query (requires viewUser action for other databases):

mongodb> db.getUsers( { usersInfo: { forAllDBs: true } } )

ClusterControl provides a nice summary of the privileges assigned to a database user. Go to Manage -> Schemas and Users -> Users and you will get a report of users' privileges, together with the advanced options like Requires SSL, Max Connections Per Hour and so on.

ClusterControl supports the privilege auditing for MySQL, MariaDB and PostgreSQL under the same user interface. 

Schema Object Auditing

Schema objects are logical structures created by users. Examples of schema objects are tables, indexes, views, routines, events, procedures, functions, triggers, and others. It basically objects that hold data or can consist of a definition only. Commonly, one would audit the permissions associated with the schema objects to detect poor security settings and understand the relation and dependencies between objects.

For MySQL and MariaDB, there are information_schema and performance_schema that we can use to basically audit the schema objects. Performance_schema is a bit depth in the instrumentation as its name suggests. However, MySQL also includes a sys schema since version 5.7.7, which is a user-friendly version of performance_schema. All of these databases are directly accessible and queryable by the clients.

Database Audit Plugins/Extensions

The most recommended way to perform statement auditing is by using an audit plugin or extension, specifically built for the database technology in use. MariaDB and Percona have their own Audit plugin implementation, which a bit different from MySQL's Audit plugin available in MySQL Enterprise. Audit records include information about the operation that was audited, the user performing the operation, and the date and time of the operation. The records can be stored in either a data dictionary table, called the database audit trail, or in operating system files, called an operating system audit trail.

For PostgreSQL, there is pgAudit, a PostgreSQL extension that provides detailed session and/or object audit logging via the standard PostgreSQL logging facility. It is basically an enhanced version of PostgreSQL's log_statement feature with the ability to easily search and lookup the captured data for auditing by following the standard audit log. 

MongoDB Enterprise (paid) and Percona Server for MongoDB (free) include an auditing capability for mongod and mongos instances. With auditing enabled, the server will generate audit messages which can be logged into syslog, console or file (JSON or BSON format). In most cases, it’s preferable to log to the file in BSON format, where the performance impact is smaller than JSON. This file contains information about different user events including authentication, authorization failures, and so on. Check out the Auditing documentation for details.

Operating System Audit Trails

It is also important to configure the operating system's audit trails. For Linux, people would commonly use auditd. Auditd is the user-space component of the Linux Auditing System and responsible for writing audit records to the disk. Viewing the logs is done with the ausearch or aureport utilities. Configuring the audit rules is done with the auditctl utility, or by modifying the rule files directly.

The following installation steps are our common practice when setting up any kind of servers for production usage:

$ yum -y install audit # apt install auditd python
$ mv /etc/audit/rules.d/audit.rules /etc/audit/rules.d/audit.rules.ori
$ cd /etc/audit/rules.d/
$ wget https://gist.githubusercontent.com/ashraf-s9s/fb1b674e15a5a5b41504faf76a08b4ae/raw/2764bf0e9bf25418bb86e33c13fb80356999d220/audit.rules
$ chmod 640 audit.rules
$ systemctl daemon-reload
$ systemctl start auditd
$ systemctl enable auditd
$ service auditd restart

Note that the last line service auditd restart is compulsory because audit doesn't work really well when loading rules with systemd. However, systemd is still required to monitor the auditd service. During startup, the rules in /etc/audit/audit.rules are read by auditctl. The audit daemon itself has some configuration options that the admin may wish to customize. They are found in the auditd.conf file.

The following line is an output taken from a configured audit log:

$ ausearch -m EXECVE | grep -i 'password' | head -1
type=EXECVE msg=audit(1615377099.934:11838148): argc=7 a0="mysql" a1="-NAB" a2="--user=appdb1" a3="--password=S3cr3tPassw0rdKP" a4="-h127.0.0.1" a5="-P3306" a6=2D6553484F5720474C4F42414C205641524941424C4553205748455245205641524941424C455F4E414D4520494E20282776657273696F6E272C202776657273696F6E5F636F6D6D656E74272C2027646174616469722729

As you can see from the above, it is easy to spot a cleartext password for MySQL ("--password=S3cr3tPassw0rdKP") using ausearch utility as captured by the auditd. This kind of discovery and auditing is vital to secure our database infrastructure, where a cleartext password is unacceptable in a secure environment.

Final Thoughts

Audit log or trail is a vital aspect that commonly is overlooked by DevOps engineers when managing infrastructures and systems, let alone the database system which is a very critical system to store sensitive and confidential data. Any exposure or breaches of your private data can be extremely damaging for the business and no one would want that to happen in the current information technology era.