Synchronizing the same set of data across multiple servers is a common practice that is followed to ensure the availability of data across various servers. Data Replication is the process of storing data in more than one site or node.

Why do you need Replication?

Replication provides redundancy and increases data availability. With multiple copies of data on different database servers, replication provides a level of fault tolerance against the loss of a single database server.

MongoDB Replica Set

MongoDB Replica Set is a group of MongoDB processes known as mongod instances that basically host the same data set. It is featured by one primary node, several secondary nodes for bearing data and optionally one arbiter node.

You can learn more about MongoDB Replica Set here

Deploying on Google Cloud Platform (GCP)

Let's begin with the deployment. We will be using Google Cloud's Compute Engine for creating VM Instances and the official MongoDB server binaries. We will be deploying 3 nodes (Primary + Secondary + Arbiter) on Compute Engine.

Picture Credits: docs.mongodb.com

Prerequisites

• Basic knowledge of Linux Terminal and DevOps
• Google Cloud Project: GCP provides a free 300$ trial for everyone signing up the first time
• Domain Name control panel: This is optional but recommended for easily connecting to the Replica Set from public networks.

The Metal

We need a minimum of 3 VM instances for Replica Set. Two of the VM's serve as Primary and Secondary nodes for data replication and third is the arbiter node. It is up to you to set the capacity for the VM's instances. For arbiter node, you can save a few bucks by setting low capacity (even GCP's free f1-micro instance gets the job done).

This article demonstrates deployment process for Primary + Secondary + Arbiter nodes but you can use the same process for deploying another Secondary node instead of Arbiter.

MongoDB's default port

MongoDB defaults to 27017 port for connection. For our VM's to open this port we need to create a Firewall Rule.

In GCP console, Go to Firewall Rule under VPC Network and create a mongo-default-port rule with the following config.

Primary and Secondary Node

We are now ready to launch our Virtual Machines. In this PSA configuration, it is recommended to keep Primary and Secondary nodes of the same config, as in any point of time these two nodes can interchange Primary and Secondary positions. For the Arbiter node, you can opt-in for a low resource VM. In regards to Storage, it is recommended to use SSD. For more information about Hardware considerations read this

Here's an example VM for one of the primary nodes.

• You can choose any Linux distro but Debian is recommended for its stability and bloatware-free environment.
• Make sure you untick HTTP/HTTPS ports (unless you have a specific use case) and add mongo-default-port network tag which we created in the last step.
• When working with stateful VM's always attach an external disk to store your data, which will allow you to retain the data when upgrading/migrating VM's and have a backup service like GCP Snapshot Schedule.

If you want to configure your VM for performance, now is a good time. One of the recommended steps for Linux server is Disable Transparent Huge Pages (THP). You can find a few more performance tweaks on the internet but be careful because every linux distro behaves differently.

Setting up MongoDB Replicas

You need to replicate the following process on all 3 VM's

Install MongoDB

Here's the official guide on installing MongoDB on Linux. Feel free to use any other community guides as long as it's downloading binaries from the official source.

Database path (Optional)

By default, MongoDB will store the data in /var/lib/mongodb. If you wish to change the directory or if you are using external disk then you need to create a directory for that path.

sudo mkdir -p /mnt/disks/persistent/db
sudo chmod 777 /mnt/disks/persistent/db

Authentication

We will be using keyfile authentication for this guide, but for a Production environment, you should consider using more secure x.509 certificates.

The keyfile contains a password/secret in plain-text. You can choose any password or generate a complex random string to store in the keyfile. We are placing the key inside /var/tmp so it is accessible to all users.

Use nano/vim to create the keyfile and type in your password/secret text.

nano /var/tmp/mongo-keyfile

Assign permission and ownership to keyfile:

sudo chmod 400 /var/tmp/mongo-keyfile
sudo chown -R mongodb:mongodb /var/tmp/mongo-keyfile

Configuration

MongoDB uses /etc/mongod.conf as startup configuration for mongod process. We can configure that file to support our Replica Set. You can learn more about mongod.conf here.

Below is the configuration file we will use in this guide.

# mongod.conf

# Storage: Where and how to store data
# Default dbPath: /var/lib/mongodb
storage:
  dbPath: <path-to-database-dir>
  journal:
    enabled: true

# Authorization with keyfile
security:
  keyFile: <path-to-keyfile>

# Logging
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Network Interfaces
# Allows access from all IP address
net:
  port: 27017
  bindIp: 0.0.0.0
  bindIpAll: true

# Replication config
replication:
   replSetName: dev-rs0
   enableMajorityReadConcern: false

# How the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

# Cloud config for enabling free monitoring
cloud:
   monitoring:
      free:
         state: on
         tags: [dev-node-0]

Set storage.dbPath, security.keyFile and replication.replSetName for your mongod.conf.

Now, restart your MongoDB deamon (mongod) to let it start with latest mongod.conf.

sudo service mongod restart

Verify that the deamon has started successfully.

sudo service mongod status

If the mongod process crashes then check out the debugging section at the bottom of the article.

Networking

VM Mapping

We will be connecting our Replica Set with hostnames, this allows us to connect to our replica set using External or Internal IP.

We need to map our Virtual Machine's /etc/hosts file to each other with the hostnames pointing to other VM's Internal IP.

10.128.0.7    mongo-node0.example.com  # Current Machine
10.128.0.8    mongo-node1.example.com  # Secondary Node
10.128.0.9    mongo-node2.example.com  # Arbiter Node

Similarly, you need to edit the /etc/hosts file on the rest of the VM's as well. You can even use the local IP 127.0.0.1 instead the Internal IP for the VM that you are currently logged into (current machine).

Domain Mapping

If you have a registered domain then you need to edit the DNS records to add A records pointing to VM's External IP. This will allows us to connect our Replica Set from the public internet.

If you don't have a registered domain or If you wish to not make the replica set available on the public internet then use any arbitrary domain name as hostname and on any system that you are using to connect to your replica set, edit the system's hosts file to point the hostnames to VM's IP.

If you are trying to connect any other VM (deployed on GCP) to your Replica Set then you can still use the VM's Internal IP to point the hostname. But for devices that will be accessing the Replica Set from outside of GCP network (like your local developer machine) then use the VM's External IP to point the hostname:

35.247.10.47      mongo-node0.example.com  # Primary Node
35.247.199.41     mongo-node1.example.com  # Secondary Node
35.82.190.33      mongo-node2.example.com  # Arbiter Node

Initiating Replica Set

Its time for us to initiate Replica Set and connect them to each other. SSH into the VM that you want as Primary node and connect to the running mongod.

mongo

Initiate the Replica Set on this node to make this a Primary node.

rs.initiate()

Check the replica status with

rs.status()

If the replica set has successfully been initialized then you'll see the replica set name and an entry in members array with the current machine state as PRIMARY.

Once the node is in PRIMARY mode, you will lose access to any administrative commands used inside MongoDB. So lets create an user with DB Admin priviledges on the Mongo's default admin database.

use admin

and create the user with admin roles.

db.createUser(
  {
    user: "username",
    pwd: "password",
    roles: [
      {
        role: "readWriteAnyDatabase",
        db: "admin"
      },
      {
        role: "userAdminAnyDatabase",
        db: "admin"
      },
      {
        role: "dbAdminAnyDatabase",
        db: "admin"
      },
      {
        role: "clusterAdmin",
        db: "admin"
      }
    ]
  }
)

The above list of roles allows almost all the permissions that any admin might need but you can configure the roles as per your requirement. It is mandatory to keep atleast one user with clusterAdmin role to handle the Replica Set.

Now exit and reconnect to mongod with user credentials

mongo -u username

you will be prompted to enter password. Now you can use any regular database commands. Whatever data you create now will be replicated by the nodes that we will add in the next step.

When you initiate the Replica Set it will be created with a default Replica configuration . You can check the configuration with

rs.conf()

Adding other nodes to Replica Set

Setting the priority for members

We need to edit the default Replica config for setting the priority of our current primary node. Priority can be any integer. I'll recommend setting higher priority for the member you want as the first preference for Primary Node voting. In this case, we are going to set priorities as this:

• 10 - Primary Node
• 5 - Secondary Node
• 0 - Arbiter Node

As replica set currently only has one primary node connected, we will change its priority by reconfiguring the rs.conf()

Save the current configuration in a variable

cfg = rs.conf()

Make sure you have 1 node present in members array and then set its new priority just like you would do in any other programming language.

cfg.members[0].priority = 10

Now to make the changes take effect, reconfigure the Replica set with new configuration

rs.reconfig(cfg)

Connecting secondary node

Before proceeding to connect the secondary node, make sure its mongod process is properly running.

Now from the PRIMARY node, add secondary node configuration.

rs.add( { host: "mongo-node1.example.com:27017", priority: 5, votes: 1 } )

After exchanging a couple of heartbeats and syncing the data, the node should attain SECONDARY status in rs.status().

Connecting arbiter node

Similar to secondary node, we will now add arbiter node.

rs.add( { host: "mongo-node2.example.com:27017", priority: 0, votes: 1, arbiterOnly: true, hidden: true } )

This node won't replicate any data and will be hidden to any read requests.

...and that's about it. You should now have a fully functional MongoDB Replica Set deployed on Google Cloud Platform.

Connecting to Replica Set from clients

There are various ways you can connect to the replica set but make sure you use the specified domain names of nodes and the replica set name that was specified in mongod.conf.

From MongoDB 3.4 onwards, you can use the Mongo Connection String in shell command, example:

mongo "mongodb://username:password@mongo-node0.example.com:27017,mongo-node1.example.com:27017,mongo-node2.example.com:27017/?replicaSet=repl-set-name"

You will be using the same string in various client drivers like Node.js, Java and Go.

Debugging

mongod process crashing

You can find the logs for crashes in the file specified at systemLog.path in mongod.conf. By default the path will be /var/log/mongodb/mongod.log. More info about log messages.

Replica Nodes not able to connect to each other

• Use ping to test the general network connectivity of each node to other nodes.
• Verify your hosts file for correctly mapping between IP and Domain Name.
• Make sure you have correct domain names as members.host in rs.conf(). Reconfigure if you dont.

Replica Set Voting

If you want to test the voting process of Replica Set then there's a helper method rs.stepDown() for switching the primary node. This method, when called on Primary Node, will make the node forgo its Primary status and trigger a vote between Replica Sets.

In the previous part, I covered Introduction and setting up Apollo GraphQL for Android and we also finished hitting our first GraphQL query. This means we received the data from the server. Now what about updating the data on the server? This resembles the POST request for REST API's. But in GraphQL world it's called Mutation, using which we can update the data on the server.

Mutation

Now let's create the similar .graphql file for our Mutation.

mutation UpdateUser(
    $userId: String!
    $name: String
    $phone: String
    $email: String
    $age: Int
) {
    updateUserDetails(
        userId: $userId
        name: $name
        phone: $phone
        email: $email
        age: $age
    ) {
        userId
    }
}

The above mutation is straightforward and can be performed with similar method as query. But usually the User entity will have many more fields and it will be defined in a Custom Data type. So our mutation might look like:

mutation UpdateUser($user: UserInput!) {
    updateUserDetails(
        user: $user
    ) {
        user {
            _id
            name
        }
    }
}

You might be wondering on how to pass objects with custom data types (like UserInput in this case) in the parameters. Well, as the Apollo generate classes for you, it also provides a builder() function to let you instantiate those input classes with data.

val userInput = UserInput.builder()
    .userId("user-id-string")
    .name("John Doe")
    .phone("987654321")
    .email("hello@world.com")
    .age(21)
    .build()

Now you can pass this object in our mutation parameter.

apolloClient.mutate(
    UpdateUserMutation.builder()
        .user(userInput)
        .build()
)
    .enqueue(object : ApolloCall.Callback<UpdateUserMutation.Data>() {

        override fun onResponse(response: Response<UpdateUserMutation.Data>) {
            if (!response.hasErrors()) {
                // Response successful
                Log.d(TAG, "Response: ${response.data()}")
            } else {
                // Request was successful but contains errors
                Log.d(TAG, "Response has errors: ${response.errors()}")
            }
        }

        override fun onFailure(e: ApolloException) {
            // Request Failed
            e.printStackTrace()
        }
    })

And there you go, we have our first mutation done.

Custom Type Adapter

There are times when data types at server side and client side won't be compatible. We can make use of Custom Type Adapters to tackle that issue. Below is the example of server accepting Date as a 'String' in UTC Format. The CustomTypeAdapter will have encode() and decode() functions for you to override and add your own parsing logic.

class CustomDateAdapter : CustomTypeAdapter<Date> {

    companion object {
        const val DATE_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
    }

    override fun encode(value: Date): CustomTypeValue<*> {
        // Parse Date in UTC TimeZone
        val calendar = Calendar.getInstance()
        calendar.timeZone = TimeZone.getTimeZone("UTC")
        calendar.time = value
        val time = calendar.time

        // Parse Date in UTC Format
        val sdf = SimpleDateFormat(DATE_FORMAT, Locale.ENGLISH)
        return CustomTypeValue.GraphQLString(sdf.format(time))
    }

    override fun decode(value: CustomTypeValue<*>): Date {
        // Parse UTC formatted Date String in Date
        val sdf = SimpleDateFormat(DATE_FORMAT, Locale.ENGLISH)
        sdf.timeZone = TimeZone.getTimeZone("UTC")
        val date = sdf.parse(value.value.toString())

        // Change timezone to default
        val calendar = Calendar.getInstance()
        calendar.timeZone = TimeZone.getDefault()
        calendar.time = date

        return calendar.time
    }
}

Now to actually make the CustomTypeAdapter work, add the CustomTypeAdapter into our ApolloClient instantiation.

ApolloClient.builder()
    .serverUrl("http://localhost:8080/graphql")
    .okHttpClient(okHttpClient)
    .addCustomTypeAdapter(CustomType.DATE, CustomDateAdapter())
    .build()

And then specify on which data type you have mapped the CustomTypeAdapter in your app level build.gradle file

android {
    ...
    apollo {
        customTypeMapping = ["Date": "java.util.Date"]
    }
    ...
}

File Upload Support

Since version 1.0.1, Apollo has added native support for uploading file on Android. It is based on this specification for backend GraphQL server.

To start with File uploading, you need to add customTypeMapping to app level build.gradle file, similar to CustomTypeAdapter:

apollo {
  customTypeMapping = [
    "Upload" : "com.apollographql.apollo.api.FileUpload"
  ]
}

GraphQL schema uses custom scalar type named Upload for FileUpload.

mutation UploadFile(
    $file: Upload!
    $type: Int!
) {
    uploadFile(
        file: $file
        type: $type
    )
}

and to call the mutation with file upload:

val file: File = ...
// Get the MIME Type for file or else return image/png as default
val mediaType = MediaType.parse(file.getMimeType() ?: "image/png")

val apolloCall = UploadFileMutation.builder()
    .file(FileUpload(mediaType.toString(), file))
    .type(2)
    .build()

// Enqueue the call however you like it

GraphQL also allows you to upload multile files in single mutation inside an array or separate input fields, if your API accepts it.

Rx Support

It's pretty straight forward to convert Callbacks into RxJava's Reactive streams. All you have to do is add Rx2 extension dependency:

implementation 'com.apollographql.apollo:apollo-rx2-support:x.y.z'

and then wrap the query into Rx2Apollo wrapper:

val observable = Rx2Apollo.from(
    apolloClient.query(
        GetAttendeeDetailsQuery.builder()
            .userId(userId)
            .eventId(eventId)
            .build()
    )
)
    .subscribeOn(Schedulers.io())
    .observeOn(AndroidSchedulers.mainThread())

Note: Apollo is soon dropping the support for RxJava1

For more information on how to use the Observables you can refer many other blog posts like this and this.

Coroutines Support

Apollo supports Coroutines with simple extension functions in Kotlin. You can check out them here.

When it comes to query it's recommended to use Coroutine Channels as they emit multiple responses, (usually one from cache and other one from network).

GlobalScope.launch {
    val attendeeQuery = apolloClient.query(
        GetAttendeesQuery.builder()
            .eventId(eventId)
            .build()
        )
            .httpCachePolicy(HttpCachePolicy.CACHE_FIRST)

    val channel = attendeeQuery.toChannel()
    channel.consumeEach {
        // it.data() contains the response
    }
}

// invoke channel.cancel() in Activity/Fragment/ViewModel's onDestroy callbacks
// to avoid any memory leaks

In case of Mutations there aren't any multiple response callbacks, so using .toDeferred() is the way to go.

GlobalScope.launch {
    val mutation =  apolloClient.mutate(
        UpdateUserMutation.builder()
            .user(userInput)
            .build()
    )

    val deferred = mutation.toDeferred()
    val response = deferred.await()
}

The .toDeferred() docs reads, "Converts an ApolloCall to an Deferred. This is a convenience method that will only return the first value emitted. If the more than one response is required, for an example to retrieve cached and network response, use toChannel() instead."

Error handling in Coroutines is done using try-catch block.

GlobalScope is only used for demonstration purpose. Please use appropriate Coroutine Scopes or suspend functions in your project.

Synchronous requests with Coroutines

At times you'll need to make a synchronous request, which Apollo used to support out of the box. But has dropped the support due to many API call's returning multiple callbacks due to caching.

The .toDeferred() method can act as Synchronous requests as the .await() function will suspend until we get the response.

There's one more implementation provided by one of the community member. (ref. #606).

To use it, add this Extension function:

suspend fun <T> ApolloCall<T>.execute() = suspendCoroutine<Response<T>> { cont ->
    enqueue(object : ApolloCall.Callback<T>() {
        override fun onResponse(response: Response<T>) {
            cont.resume(response)
        }

        override fun onFailure(e: ApolloException) {
            cont.resumeWithException(e)
        }
    })
}

Now you can make synchronous requests by just calling .execute() on Apollo Call's.

GlobalScope.launch {
    try {
        val response = client.mutate(
            UpdateUserMutation.builder()
                .user(userInput)
                .build()
        ).execute()

        if (!response.hasErrors()) {
            // Response successful
            // get data in response.data()
        } else {
            // Response has errors
        }
    } catch (e: Exception) {
        // Request failed
        // It's best practice to catch various exception's in separate catch blocks
    }
}

You can use runBlocking { ... } instead of GlobalScope to return any values from inside the coroutine. This method is really useful in WorkManager's.

What is GraphQL?

"GraphQL decouples apps from services by introducing a flexible query language. Instead of a custom API for each screen, app developers describe the data they need, service developers describe what they can supply, and GraphQL automatically matches the two together. Teams ship faster across more platforms, with new levels of visibility and control over how their data is used." - apollographql.com

In simple words, GraphQL lets the client decide which data they want, rather than having the server send a fixed set of data. GraphQL is an alternative approach to building an API over REST.

Adding GraphQL to your Android project:

First of all you will need to add repository and dependency on Project level build.gradle file:

buildscript {
  repositories {
    jcenter()
  }
  dependencies {
    classpath 'com.apollographql.apollo:apollo-gradle-plugin:x.y.z'
  }
}

x.y.z represents the version number you want to add. You can find the latest version number over at Apollo-Android GitHub repo.

Now apply the plugin on App level build.gradle file:

apply plugin: 'com.apollographql.android'

This should give you the core Apollo GraphQL client. To support Apollo GraphQL on Android we need to build apollo-runtime and android-support dependencies.

implementation 'com.apollographql.apollo:apollo-runtime:1.0.0'
implementation 'com.apollographql.apollo:apollo-android-support:1.0.0'
implementation 'com.squareup.okhttp3:okhttp:3.14.2'

This provides us with support for working with Android's UI thread and Worker threads. We are building OkHttp, because Apollo GraphQL uses OkHttp as the underlying network driver.

Additionally, if you want Rx2 or Coroutines support then Apollo has you covered as well.

implementation 'com.apollographql.apollo:apollo-rx2-support:1.0.0'
implementation 'com.apollographql.apollo:apollo-coroutines-support:1.0.0'

• Replace 1.0.0 with the latest version of the libraries

There's one more step left before we get started. As Apollo handles the Database Schema on backend, we need to add schema.json in our Android project.

apollo schema:download --endpoint=http://localhost:8080/graphql schema.json

This will require you to install Apollo-CLI.

At the level of /main folder (inside your project) you need to create the following directory structure and paste the schema.json

And that's about it. We are ready to rock the GraphQL client.

Let's write our first basic query

Create the file GetUser.graphql with the following text in the same location as schema.json:

query GetUser( $userId: String!, $phone: String ) {
  getUserDetails( userId: $userId, phone: $phone ) {
    name
    email
    age
  }
}

• GetUser is the client side name for the query and getUserDetails is what the server expects as a query.
• $userId, $phone are the variables that we declare to pass down query parameters.
• 'name', 'email', 'age' are the variables that we want the query to return.

Now the actual execution of the query:

Following function builds and returns the Apollo client.

fun provideApolloClient( ): ApolloClient {
        return ApolloClient.builder()
            .serverUrl("http://localhost:8080/graphql")
            .okHttpClient(OkHttpClient().newBuilder().build())
            .build()
}

Now build your Android project which will trigger Apollo to generate all the needed classes.

Remember, Apollo also parse the response and generate classes for you. So there's no need to build your own data class and then parse response with any JSON parsing libraries.

val apolloClient = provideApolloClient( )

apolloClient.query(
            GetUserQuery.builder()
                .userId("provide-user-id")
                .phone("provide-phone")
                .build()
        )
            .enqueue(object : ApolloCall.Callback<GetUserQuery.Data>() {

                override fun onResponse(response: Response<GetUserQuery.Data>) {
                    if (!response.hasErrors()) {
                        // Here response().data() contains the data you requested
                    } else {
                        Log.d(TAG, "Request Failure ${response.errors()}")
                    }
                }

                override fun onFailure(e: ApolloException) {
                    e.printStackTrace()
                }
            })

That's about it!

Now that we have taken the first step into GraphQL world let's dive deeper into some real-world use cases into the Part II of this series.

Bonus tip: Before you are writing any new query/mutation, just update the GraphQL schema.json and see if the existing .graphql files compile or not. As GraphQL codegen all the classes at compile time and does not state which file is causing the validation failure.

While writing this article I have found this cool GraphQL plugin for Android Studio which will help you with code completion and validation errors. To setup the plugin you need to create a config file named .graphqlconfig in the same folder where you have kept schema.json and .graphql files.

{
    "name": "Demo App GraphQL Schema",
    "schemaPath": "schema.json",
    "extensions": {
        "endpoints": {
            "Default GraphQL Endpoint": {
                "url": "http://192.168.0.123:2000/graphql",
                "headers": {
                    "user-agent": "JS GraphQL"
                },
                "introspect": true
            }
        }
    }
}