Set Up Highly Available MongoDB Sharded Cluster on Ubuntu

This tutorial will show you how to set up a fault-tolerant MongoDB sharded cluster on a Ubuntu or Debian. For instance, if you would like to use remote shared storage for this sharded cluster set up, follow this tutorial.

A MongoDB sharded cluster consists of three components:

Shard that can be deployed as a replica set.
Mongos acts as a query router, providing an interface between client applications and the sharded cluster.
Config store metadata and configuration settings for the cluster.

The following illustration describes the interaction of components within a sharded cluster:

Note: This guide is specifically written for Ubuntu 18.04, 19.04, 19.10, 20.04 and Debian 9, 10.

Prerequisites

To follow this tutorial along, you will need at least 10 (physical or virtual) machines installed with Ubuntu or Debian having sudo non-root user privileges.

We will use these ten machines with the information (as described above) for our sharded cluster throughout this tutorial. Make sure you substitute, hostname, ip address, domain and red highlighted text with yours wherever applicable.

Make sure you have completed basic network settings i.e. hostname, timezone and ip addresses on each of your servers.

You can set hostname on your servers with below command:

sudo hostnamectl set-hostname server-name.domain

You can set timezone on your servers with below command:

sudo timedatectl set-timezone US/New_York

Update Hosts Files

sudo nano /etc/hosts

Add the following to the hosts file, make sure you replace the highlighted text with yours and (if possible) keep the hostnames same to avoid confusion:

# Config Server
192.168.10.1 cfg1.example.pk
192.168.10.2 cfg2.example.pk
192.168.10.3 cfg3.example.pk

# Shard Server Replica Set (rs0)
192.168.10.4 shrd1.example.pk
192.168.10.5 shrd2.example.pk
192.168.10.6 shrd3.example.pk

# Shard Server Replica Set (rs1)
192.168.10.7 shrd4.example.pk
192.168.10.8 shrd5.example.pk
192.168.10.9 shrd6.example.pk

# Mongos (Query Router)
192.168.10.10 qrt1.example.pk
192.168.10.11 qrt2.example.pk

Save and close when you are finished.

Repeat the same on each of your remaining servers before proceeding to next step.

Add MongoDB Source

You need to add MongoDB official repository in Ubuntu's source list in order to install latest release using apt package manager:

Type below to add mongodb repository:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4b7c549a058f8b6b
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list

Make sure you repeat the same steps on each of your remaining servers before proceeding to next step.

Installing MongoDB

When you are finished adding mongodb official repository on all of the servers, login to your first server (cfg1.example.pk) in our case.

Type below command to install mongodb:

sudo apt update
sudo apt -y install mongodb-org

This command installs mongodb-org, a meta-package that includes the following:

mongodb-org-server is a standard MongoDB daemon, including init scripts and configurations
mongodb-org-mongos contains MongoDB shard daemon
mongodb-org-shell used to interact with MongoDB via the command line
mongodb-org-tools contains a few basic tools to restore, import, and export data, as well as a variety of other functions.

These packages provide a solid foundation that will serve most use cases, hence we recommend installing them all. However, if you want a more minimal installation, you can selectively install packages from the above list rather than using the mongodb-org meta-package.

Make sure you repeat the same installation step on each of your remaining servers before proceeding to next step.

Create an Administrative User

Login to your first config server (cfg1.example.pk) in our case, we intend to use as the primary member of replica set of config servers.

Start mongodb service with below command:

sudo systemctl start mongod.service
sudo systemctl enable mongod.service

Access mongo shell with below command:

mongo

On mongo shell, type below to switch to the default admin database:

use admin

Create an administrative user with root privileges, make sure you replace “password” with a strong password of your choice:

db.createUser({user: "administrator", pwd: "password", roles:[{role: "root", db: "admin"}]})

This will return similar to the following output:

Successfully added user: {
        "user" : "administrator",
        "roles" : [
                {
                        "role" : "root",
                        "db" : "admin"
                }
        ]
}

Type below to exit from mongo shell:

quit()

Set Up MongoDB Authentication

In this section, you need to create a key file that will be used to secure authentication between the members of MongoDB replica set.

On (cfg1.example.pk) in our case, type below command to generate a key file:

openssl rand -base64 756 > ~/mongodb_key.pem

For this guide, we generated a key file under the home directory of user.

Copy ~/mongodb_key.pem file to /var/lib/mongodb/, and set appropriate permissions like below:

sudo cp ~/mongodb_key.pem /var/lib/mongodb/
sudo chown -R mongodb:mongodb /var/lib/mongodb/mongodb_key.pem
sudo chmod -R 400 /var/lib/mongodb/mongodb_key.pem

Next, copy mongodb_key.pem file from cfg1.example.pk to each of your remaining servers under the same location with identical permissions before proceeding to next step.

Create the Config Server Replica Set

sudo nano /etc/mongod.conf

Add, update the following highlighted values, make sure you replace default port with 27019

port: 27019
  bindIp: cfg1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

replication:
  replSetName: configReplSet

sharding:
  clusterRole: configsvr

Save and close file when you are finished.

Repeat the same on each of your remaining config servers (cfg2.example.pk, cfg3.example.pk) in our case, before proceeding to next step.

For reference, you can see below mongod.conf from cfg1.example.pk in our case.

This mongod.conf from cfg2.example.pk in our case.

and this mongod.conf from cfg3.example.pk in our case.

When you are finished with the above on all three config servers, restart the mongodb service to take changes into effect:

sudo systemctl start mongod.service
sudo systemctl enable mongod.service

Confirm that the mongodb service is active and running:

sudo systemctl status mongod.service

screenshot of cfg1.example.pk

screenshot of cfg2.example.pk

screenshot of cfg3.example.pk

Initiate the config replica set.

On your first config server, (cfg1.example.pk) in our case, connect to the MongoDB shell over port 27019 with the administrative user like below:

mongo --host cfg1.example.pk --port 27019 -u administrator --authenticationDatabase admin

This will prompt you for password:

MongoDB shell version v4.2.3
Enter password:

From the mongo shell, initiate the config server's replica set like below:

rs.initiate({ _id: "configReplSet", configsvr: true, members: [{ _id : 0, host : "cfg1.example.pk:27019"},{ _id : 1, host : "cfg2.example.pk:27019"},{ _id : 2, host : "cfg3.example.pk:27019"}]})

You will see a message similar to below indicating the operation succeeded:

Notice that the MongoDB shell prompt has also changed to configReplSet:PRIMARY> or configReplSet:SECONDARY>.

To make sure that each config server has been added to the replica set, type below on mongo shell:

rs.status()

If the replica set has been configured properly, you’ll see output similar to the following:

Now exit from mongo shell with below command:

quit()

Create the Shard Replica Sets

In this section we will configure shard replica set (rs0) on (shrd1, shrd2, shrd3) servers.

sudo nano /etc/mongod.conf

Add, update the following values, make sure you replace default port with 27018:

net:
  port: 27018
  bindIp: shrd1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

replication:
  replSetName: rs0

sharding:
  clusterRole: shardsvr

Save and close file when you are finished.

Make sure you repeat the same on (shrd2.example.pk, shrd3.example.pk) as well.

For reference you can see below screenshot of mongod.conf file on shrd1.example.pk after the changes we made.

Below screenshot of mongod.conf from shrd2.example.pk

and below screenshot of mongod.conf from shrd3.example.pk

When you are finished with the above step, start mongodb service on (shrd1, shrd2, shrd3) to take changes into effect:

sudo systemctl start mongod
sudo systemctl enable mongod
sudo systemctl status mongod

screenshot of mongod service status from shrd1.example.pk

screenshot of mongod service status from shrd2.example.pk

screenshot of mongod service status from shrd3.example.pk

When you are finished with the above step on (shrd1, shrd2, shrd3), proceed with the below.

In this step, we will configure shard replica set (rs1) on (shrd4, shrd5, shrd6) servers.

Login to (shrd4.example.pk) in our case, edit /etc/mongod.conf file:

sudo nano /etc/mongod.conf

Add, update the following values, make sure you replace default port with 27018:

net:
  port: 27018
  bindIp: shrd4.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

replication:
  replSetName: rs1

sharding:
  clusterRole: shardsvr

Save and close file when you are finished.

Make sure you repeat the same on (shrd5.example.pk, shrd6.example.pk) as well.

For reference you scan see below screenshot of mongod.conf from shrd4.example.pk after the changes we made.

Below screenshot of mongod.conf from shrd5.example.pk

and below screenshot of mongod.conf from shrd6.example.pk

When you are finished with the above step, restart the mongod service on (shrd4, shrd5, shrd6) to take changes into effect:

sudo systemctl start mongod
sudo systemctl enable mongod
sudo systemctl status mongod

mongod service status screenshot from shrd4.example.pk

mongod service status screenshot from shrd5.example.pk

mongod service status screenshot from shrd6.example.pk

Initiate the shard replica set.

Login to your first shard server (shrd1.example.pk) in our case, connect to mongo shell on port 27018 with administrative user authentication like below:

mongo --host shrd1.example.pk --port 27018 -u administrator --authenticationDatabase admin

Type below on mongo shell to initiate shard replica set (rs0):

rs.initiate({ _id : "rs0", members:[{ _id : 0, host : "shrd1.example.pk:27018" },{ _id : 1, host : "shrd2.example.pk:27018" },{ _id : 2, host : "shrd3.example.pk:27018" }]})

This will return { "ok" : 1 } indicating that shard replica set rs0 is initiated successfully.

Now exit from the mongo shell with below command:

quit()

Next, login to (shrd4.example.pk) in our case, connect to mongo shell on port 27018 with administrative user authentication like below:

mongo --host shrd4.example.pk --port 27018 -u administrator --authenticationDatabase admin

Type below on mongo shell to initiate shard replica set (rs1):

rs.initiate({ _id : "rs1", members:[{ _id : 0, host : "shrd4.example.pk:27018" },{ _id : 1, host : "shrd5.example.pk:27018" },{ _id : 2, host : "shrd6.example.pk:27018" }]})

This will return { "ok" : 1 } indicating that shard replica set rs1 is initiated successfully.

Now exit from the mongo shell with below command:

quit()

Configure Mongos (Query Router)

sudo systemctl stop mongod.service
sudo systemctl disable mongod.service

The mongos service that we will create in below step needs to obtain data locks that conflicts with mongod, so be sure mongod is stopped before proceeding:

sudo systemctl status mongod.service

As you can see mongod.service has been stopped on query router.

Type below to create /etc/mongos.conf file:

sudo nano /etc/mongos.conf

Add the following configuration directives:

# Where write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongos.log

# network interfaces
net:
  port: 27017
  bindIp: qrt1.example.pk

security:
  keyFile: /var/lib/mongodb/mongodb_key.pem

sharding:
  configDB: configReplSet/cfg1.example.pk:27019,cfg2.example.pk:27019,cfg3.example.pk:27019

Save and close file when you are finished.

You can see final mongos.conf file in below screenshot.

Next, create a systemd service unit file for mongos like below:

sudo nano /lib/systemd/system/mongos.service

Add the following parameters:

[Unit]
Description=Mongo Cluster Router
After=network.target

[Service]
User=mongodb
Group=mongodb

ExecStart=/usr/bin/mongos --config /etc/mongos.conf

LimitFSIZE=infinity
LimitCPU=infinity
LimitAS=infinity

LimitNOFILE=64000
LimitNPROC=64000

TasksMax=infinity
TasksAccounting=false

[Install]
WantedBy=multi-user.target

Save and close file when you are finished.

You can see final mongos.service file in below screenshot.

Next, start mongos.service with below command to activate query router.

sudo systemctl start mongos.service
sudo systemctl enable mongos.service

Confirm that mongos is running with below command:

sudo systemctl status mongos.service

You can see below screenshot of mongos status

Repeat the same configuration on (qrt2.example.pk) before proceeding to next step.

Add Shards to the Cluster

From (qrt1) or (qrt2), connect to mongo shell on port 27017 with administrative authentication like below:

mongo --host qrt1.example.pk --port 27017 -u administrator --authenticationDatabase admin

On mongo shell, type below to add shard replica set (rs0) in the cluster:

sh.addShard( "rs0/shrd1.example.pk:27018,shrd2.example.pk:27018,shrd2.example.pk:27018")

Type below to add shard replica set (rs1) in the cluster:

sh.addShard( "rs1/shrd4.example.pk:27018,shrd5.example.pk:27018,shrd6.example.pk:27018")

You will see the similar output indicating that shard replica set (rs0, rs1) added successfully.

At this stage, your MongoDB sharded cluster is active and running.

The last step is to enable sharding for database. This process takes place in stages due to the organization of data in MongoDB. To understand how data will be distributed, let’s quickly understand the main data structures:

Databases - The broadest data structure in MongoDB, used to hold groups of related data.
Collections - Analogous to tables in traditional relational database systems, collections are the data structures that comprise databases
Documents - The most basic unit of data storage in MongoDB. Documents use JSON format for storing data using key-value pairs that can be queried by applications

Sharding Strategy

Before you enable sharding, you’ll need to decide on a sharding strategy. When data is distributed among the shards, MongoDB needs a way to sort it and know which data is on which shard. To do this, it uses a shard key, a designated field in your documents that is used by the mongos query router know where a given piece of data is stored.

The two most common sharding strategies are:

Range-based sharding divides your data based on specific ranges of values in the shard key.
Hash-based sharding distributes data by using a hash function on your shard key for a more even distribution of data among the shards.

This is not intended to be a comprehensive explanation to choosing a sharding strategy. You may wish to consult with official resource of MongoDB’s documentation on sharding strategy.

Enable Sharding

Now that we’ve decided to use a hash-based sharding strategy, we'll enable sharding at the collections level. This allows the documents within a collection to be distributed among your shards.

mongo --host qrt1.example.pk --port 27017 -u administrator --authenticationDatabase admin

From the mongos shell, type below to create a test database called testDB

use testDB

Create a new collection called testCollection and hash its _id key. The _id key is already created by default as a basic index for new documents:

db.testCollection.ensureIndex( { _id : "hashed" } )

Type below to enable sharding for newly created database:

sh.enableSharding("testDB")
sh.shardCollection( "testDB.testCollection", { "_id" : "hashed" } )

This enables sharding across any shards that you added to your cluster in the Add Shards to the Cluster step.

To verify that the sharding was successfully enabled, type below to switch to the config database:

use config

Next, run below method:

db.databases.find()

This will return a list of all databases with useful information similar to like below:

In the above output, you can see only one entry for the testDB database we just created.

Once you enable sharding for a database, MongoDB assigns a primary shard for that database where MongoDB stores all data in that database.

Verify Shard Cluster

To ensure your data is being distributed evenly in the testDB database and collection you configured above, you can follow these steps to generate some basic test data and see how it is divided among the shards.

Connect to the mongos shell on your query router:

mongo --host qrt1.example.pk --port 27017 -u administrator -p --authenticationDatabase admin

Switch to your testDB database:

use testDB

Type the following code in the mongo shell to generate 10000 simple documents and insert them into testCollection:

for (var i = 1; i <= 10000; i++) db.testCollection.insert( { x : i } )

Check your data distribution:

db.testCollection.getShardDistribution()

This will return information similar to the following:

The sections beginning with Shard give information about each shard in your cluster. Since we only added 2 shards with three members each, there are only two sections, but if you add more shards to the cluster, they’ll show up here as well.

The Totals section provides information about the collection as a whole, including its distribution among the shards. Notice that distribution is not perfectly equal. The hash function does not guarantee absolutely even distribution, but with a carefully chosen shard key it will usually be fairly close.

When you’re finished, we recommend you to delete the testDB (because it has no use) with below command:

use testDB
db.dropDatabase()

Wrapping up

Now that you have successfully deployed a highly available fault-tolerant MongoDB sharded cluster ready to use for your production environment, it is recommended to configure a firewall to limit ports 27018 and 27019 to only accept traffic between hosts within your cluster.