Oracle Linux/MySQL

Simple Cluster configuration

Will consist of 5 components:

Management Node

holds the configuration for MySQL Cluster in a file called config.ini. It also writes a cluster log, and takes part in arbitration to prevent split-brain or network partitioning.

(2) Data Nodes

stores the data and the tables. The data nodes takes cares of managing the transactions, recovery, etc

(2) SQL Nodes

the most common way to read/write data to the Data Nodes is by using the MySQL Server (aka SQL Node)

Management Node Configuration

dependencies

There are some packages you may need to install, but are not required. More info can be found here.

~$ sudo yum install -y wget telnet vim net-tools bind-utils

While most of the guides I found perform installation directly using the RPM packages, I prefer to install packages from a repository whenever available. MySQL provides a YUM repository.

~$ wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
~$ sudo yum localinstall -y mysql80-community-release-el7-3.noarch.rpm

Verify you can see the new cluster packages and search them.

~$ yum repolist all | grep mysql
~$ yum --enablerepo=mysql-cluster-8.0-community search mysql

Enable the cluster repo by setting enabled=1

~$ sudo vim /etc/yum.repos.d/mysql-community.repo
...
[mysql-cluster-8.0-community]
name=MySQL Cluster 8.0 Community
baseurl=http://repo.mysql.com/yum/mysql-cluster-8.0-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

Do a quick check-update to refresh the package list cache

~$ sudo yum check-update

We also need to open a port and make sure our networking interface is setup correctly. In my environment, servers communicate with each other through a virtual 10GB network rather than traversing the physical 1GB network.

~$ sudo firewall-cmd --permanent --zone=internal --change-interface=eth1
~$ sudo firewall-cmd --zone=internal --permanent --add-port=1186/tcp
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --zone=internal --list-all

package installation

There are a couple of dependencies we need to install.

~$ sudo yum install -y epel-release && sudo yum install -y perl-Class-MethodMaker

Install the packages

 ~$ sudo yum install -y mysql-cluster-community-management-server mysql-cluster-community-client mysql-cluster-community-libs

configure management node cluster

Now we can setup a basic configuration file for the cluster.

~$ sudo mkdir -p /var/lib/mysql-cluster
~$ sudo vim /var/lib/mysql-cluster/config.ini
...
# TCP PARAMETERS
[tcp default]SendBufferMemory=2M
ReceiveBufferMemory=2M

# MANAGEMENT NODE PARAMETERS
[ndb_mgmd default]
# Directory for MGM node log files
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
#Management Node db1
HostName=rncv-sqlmgmt01.virtual.local

# DATA NODE PARAMETERS
[ndbd default]
NoOfReplicas=2      # Number of replicas
DataMemory=256M     # Memory allocate for data storage
IndexMemory=128M    # Memory allocate for index storage
#LockPagesInMainMemory=1 # Locks data node processes into memory  ##causes crash
#Directory for Data Node
DataDir=/var/lib/mysql-cluster

#NoOfFragmentLogFiles=300 ##causes crash
MaxNoOfConcurrentOperations=100000
#SchedulerSpinTimer=400  ##causes crash
#SchedulerExecutionTimer=100  ##causes crash
#RealTimeScheduler=1  ##causes crash
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
RedoBuffer=32M
MaxNoOfTables=1024
MaxNoOfOrderedIndexes=256

[ndbd]
#Data Node db2
HostName=rncv-sqldata01.virtual.local
#LockExecuteThreadToCPU=1  ##causes crash
#LockMaintThreadsToCPU=0  ##causes crash

[ndbd]
#Data Node db3
HostName=rncv-sqldata02.virtual.local
#LockExecuteThreadToCPU=1  ##causes crash
#LockMaintThreadsToCPU=0  #causes crash

[mysqld]
#SQL Node db4
HostName=rncv-sql01.virtual.local

[mysqld]
#SQL Node db5
HostName=rncv-sql02.virtual.local

[mysqld]

[api]

Now we can implement the config

~$ sudo ndb_mgmd --config-file=/var/lib/mysql-cluster/config.ini
MySQL Cluster Management Server mysql-8.0.19 ndb-8.0.19
2020-02-24 17:54:57 [MgmtSrvr] INFO     -- The default config directory '/usr/mysql-cluster' does not exist. Trying to create it...
2020-02-24 17:54:57 [MgmtSrvr] INFO     -- Sucessfully created config directory
2020-02-24 17:54:57 [MgmtSrvr] WARNING  -- at line 12: [DB] IndexMemory is deprecated, will use Number bytes on each ndbd(DB) node allocated for storing indexes instead

And finally we can view the implementation:

~$ ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	2 node(s)
id=2 (not connected, accepting connect from rncv-sqldata01.virtual.local)
id=3 (not connected, accepting connect from rncv-sqldata02.virtual.local)

[ndb_mgmd(MGM)]	1 node(s)
id=1	@10.10.0.40  (mysql-8.0.19 ndb-8.0.19)

[mysqld(API)]	2 node(s)
id=4 (not connected, accepting connect from rncv-sql01.virtual.local)
id=5 (not connected, accepting connect from rncv-sql02.virtual.local)

ndb_mgm> quit

Data Node Configuration

The data node configuration is very similar to the Management node configuration.

dependencies

Again, same as with the management node configuration, there are some packages you may need to install, but are not required. More info can be found here.

~$ sudo yum install -y wget telnet vim net-tools bind-utils

While most of the guides I found perform installation directly using the RPM packages, I prefer to install packages from a repository whenever available. MySQL provides a YUM repository.

~$ wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
~$ sudo yum localinstall -y mysql80-community-release-el7-3.noarch.rpm

Verify you can see the new cluster packages and search them.

~$ yum repolist all | grep mysql
~$ yum --enablerepo=mysql-cluster-8.0-community search mysql

Enable the cluster repo by setting enabled=1

~$ sudo vim /etc/yum.repos.d/mysql-community.repo
...
[mysql-cluster-8.0-community]
name=MySQL Cluster 8.0 Community
baseurl=http://repo.mysql.com/yum/mysql-cluster-8.0-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

Do a quick check-update to refresh the package list cache

~$ sudo yum check-update

We also need to open a port and make sure our networking interface is setup correctly. In my environment, servers communicate with each other through a virtual 10GB network rather than traversing the physical 1GB network.

~$ sudo firewall-cmd --permanent --zone=internal --change-interface=eth1
~$ sudo firewall-cmd --zone=internal --permanent --add-port=32768-60999/tcp
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --zone=internal --list-all

package installation

There are a couple of dependencies we need to install.

~$ sudo yum install -y epel-release && sudo yum install -y perl-Class-MethodMaker perl-Data-Dumper

Install the packages

~$ sudo yum install -y mysql-cluster-community-data-node
~$ sudo yum install -y mysql-cluster-community-client mysql-cluster-community-libs

configure data node

We now need to tell the data node where to find the sql management node.

~$ sudo vim /etc/my.cnf
...
[mysqld]
ndbcluster
ndb-connectstring=rncv-sqlmgmt01.virtual.local     # IP address of Management Node
 
[mysql_cluster]
ndb-connectstring=rncv-sqlmgmt01.virtual.local     # IP address of Management Node

Create a folder for the configuration files and data.

~$ sudo mkdir -p /var/lib/mysql-cluster

I had an issue starting the ndbd client because of the folder permissions set on the folder above. Since ndbd runs as whatever user it is launched from, make sure that user has read, write, and execute permissions to the folder.

start data node

Now we can start the data node and confirm it's connected.

~$ ndbd
2020-03-04 13:35:04 [ndbd] INFO     -- Angel connected to 'rncv-sqlmgmt01.virtual.local:1186'
2020-03-04 13:35:04 [ndbd] INFO     -- Angel allocated nodeid: 2

From the management node you can also confirm this:

#management_node~$ ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	2 node(s)
id=2	@10.10.0.41  (mysql-8.0.19 ndb-8.0.19, starting, Nodegroup: 0)
id=3 (not connected, accepting connect from rncv-sqldata02.virtual.local)

[ndb_mgmd(MGM)]	1 node(s)
id=1	@10.10.0.40  (mysql-8.0.19 ndb-8.0.19)

[mysqld(API)]	2 node(s)
id=4 (not connected, accepting connect from rncv-sql01.virtual.local)
id=5 (not connected, accepting connect from rncv-sql02.virtual.local)

If you have a 2nd data node, repeat this process

SQL Node Configuration

dependencies

Again, same as with the management node configuration, there are some packages you may need to install, but are not required. More info can be found here.

~$ sudo yum install -y wget telnet vim net-tools bind-utils

While most of the guides I found perform installation directly using the RPM packages, I prefer to install packages from a repository whenever available. MySQL provides a YUM repository.

~$ wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
~$ sudo yum localinstall -y mysql80-community-release-el7-3.noarch.rpm

Verify you can see the new cluster packages and search them.

~$ yum repolist all | grep mysql
~$ yum --enablerepo=mysql-cluster-8.0-community search mysql

Enable the cluster repo by setting enabled=1

~$ sudo vim /etc/yum.repos.d/mysql-community.repo
...
[mysql-cluster-8.0-community]
name=MySQL Cluster 8.0 Community
baseurl=http://repo.mysql.com/yum/mysql-cluster-8.0-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

Do a quick check-update to refresh the package list cache

~$ sudo yum check-update

We also need to open a port and make sure our networking interface is setup correctly. In my environment, servers communicate with each other through a virtual 10GB network rather than traversing the physical 1GB network.

~$ sudo firewall-cmd --permanent --zone=internal --change-interface=eth1
~$ sudo firewall-cmd --permanent --zone=internal --add-service=mysql
~$ sudo firewall-cmd --permanent --zone=internal --add-port=33060/tcp
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --zone=internal --list-all

package installation

There are a couple of dependencies we need to install.

~$ sudo yum install -y epel-release && sudo yum install -y perl-Class-MethodMaker perl-Data-Dumper

Install the packages

~$ sudo yum install -y mysql-cluster-community-server
~$ sudo yum install -y mysql-cluster-community-client mysql-cluster-community-libs mysql-cluster-community-common

configure sql node

We now need to tell the sql node where to find the sql management node. When I did this, my.cnf already existed and had a [mysqld] section.

~$ sudo vim /etc/my.cnf
...
[mysqld]
ndbcluster
ndb-connectstring=rncv-sqlmgmt01.virtual.local       # IP address for server management node
default-storage-engine=ndbcluster     # Define default Storage Engine used by MySQL
 
[mysql_cluster]
ndb-connectstring=rncv-sqlmgmt01.virtual.local      # IP address for server management node

I also had to configure SELinux as it was blocking mysqld from connecting to the sql management node. It is best to follow the steps listed here, but ultimately you'll just be creating a policy allowing mysqld the ability to create tcp socket connections over ephemeral ports. It should look something like this:

$ cat local-mysql1.te

module local-mysql1 1.0;

require {
	type ephemeral_port_t;
	type mysqld_t;
	class tcp_socket name_connect;
}

#============= mysqld_t ==============

#!!!! This avc can be allowed using one of the these booleans:
#     nis_enabled, mysql_connect_any
allow mysqld_t ephemeral_port_t:tcp_socket name_connect;

start sql node

~$ sudo systemctl restart mysqld.service

If you have a 2nd sql node, repeat this process