MongoDB Stitch Mobile Sync – The AWS re:Invent Stitch Rover Demo

At AWS re:Invent, we demonstrated how MongoDB Mobile, MongoDB Stitch, and AWS services could be used to build a cloud-controlled Mars rover – read the overview post for the setup. This post focuses on how the mission control app records the user commands in MongoDB Atlas.

Mission Control is a simple web application that we ran on an iPad. The web app takes commands from the user through its UI which displays directions for the rover to travel in. Each command sets the rover off in that direction for a short fixed amount of time.

MongoDB Stitch QueryAnywhere rover Mission Control web app

Rather than sending the command directly to the rover, and as Mars is a long way away and network connections are not always reliable, the app stores the commands in an array within the rover’s document in Atlas – in that way, commands can be queued up and sent to the rover as soon as it’s back online.

MongoDB Stitch QueryAnywhere

This is what the document looks like after a few commands have been submitted (but not yet acted on by the rover):

{
    "_id" : "5bee1053fdc728f2623e20eb",
    "moves" : [
        {
            "_id" : "5c0e4db5119f6e36c7d06f55",
            "angle" : 90,
            "speed" : 2
        },
        {
            "_id" : "5c0e4dbc119f6e36c7d06f56",
            "angle" : 118,
            "speed" : 3
        },
        {
            "_id" : "5c0e4dc3119f6e36c7d06f57",
            "angle" : 45,
            "speed" : -1
        }
    ],
    "__stitch_sync_version" : {
        "spv" : 1,
        "id" : "2f704b04-2338-4c75-a7cf-d555c94cb556",
        "v" : NumberLong(10313)
    }
}

A traditional way of access the database from the frontend app would be to stand up an app server, implement a custom REST API and data access control layer, and then send the commands to it from the frontend app. MongoDB Stitch massively simplifies that workflow by letting a web (or mobile) app execute MongoDB Query Language operations directly – removing the need for thousands of lines of boilerplate code.

When we explained this access model to demo visitors, some were nervous about this approach as we’ve been taught that the schema and database access should be hidden from the frontend (remember SQL injection attacks?). Fortunately, Stitch QueryAnywhere is made up of two components – the first is the Stitch SDK that enables direct access to the database, the second is the sophisticated, fine-grained user access controls provided by the Stitch service. If that doesn’t allay your fears then you have the option to obfuscate the schema by accessing the database through Stitch Functions.

Updating the document from the web app is a cinch with Stitch. The first step is to import the Stitch SDK:

<head>
  <script src="https://s3.amazonaws.com/stitch-sdks/js/bundles/4.0.15-0/stitch.js"></script>
  <script src="https://s3.amazonaws.com/stitch-sdks/bson/bson.bundle.js"></script>
</head>

The second step is to create the object/document for the command and push it onto the moves array within this rover’s document in the rover.rovers collection:

function pushMoveToRover(roverId) {
    const client = stitch.Stitch.defaultAppClient;
    const coll = client.getServiceClient(stitch.RemoteMongoClient.factory,
        'mongodb-atlas').db('rover').collection('rovers');
    const angle = parseInt($("#angle").val());
    const speed = parseInt($("#speed").val());
    if (isNaN(angle) || isNaN(speed)) {
        return;
    }
    const move = {"_id": new BSON.ObjectID().toHexString(), angle, speed};
    coll.updateOne({"_id": roverId}, {"$push": { "moves": move }, 
        "$inc": {"__stitch_sync_version.v": 1}});
}

If you are used to working with the MongoDB Query Language from an app server, that should seem very familiar. The only thing that might catch your eye is that __stitch_sync_version.v. That’s part of how this update will get to the MongoDB Mobile database embedded in the rover. We’ll explain that in the next part.

If you can’t wait then you can find all of the code in the Stitch Rover GitHub repo.





MongoDB Stitch/Mobile Mars Rover Lands at AWS re:Invent

Powered by MongoDB Mobile, MongoDB Stitch, and AWS Kinesis, the MongoDB rover debuted at AWS re:Invent. More than a thousand people stopped by our demo to see if they could navigate the rover around a treacherous alien landscape.

MongoDB Stitch & Rover at AWS re:Invent

One of the challenges in controlling a rover on the other side of the Solar System is that there will frequently be times when you lose connectivity. To solve that, we store all of the commands in MongoDB Atlas and sync them to the rover (in its local MongoDB Mobile database) whenever it’s contactable.

MongoDB Stitch Rover Application Architecture

Visitors controlled the rover via a JavaScript web app, which uses Stitch QueryAnywhere to write commands for the rover directly to a collection in MongoDB Atlas (the fully managed hosted MongoDB database, in this case, running in the AWS cloud).

After the commands are sent to the cloud, Stitch Mobile Sync takes over, ensuring all commands are automatically written to the MongoDB Mobile database embedded on the rover (running Android Things on a Raspberry Pi 3b) by Stitch Mobile Sync. The Java app on the rover then acts on the synced commands (moving the Rover), before removing them from the mobile database to reflect that it’s completed with that set of commands – with the updated status then finally synced back to Atlas by Stitch Mobile Sync.

The rover can also record environmental sensor data in MongoDB Mobile, which Stitch Mobile Sync syncs to the sensors collection in Atlas. A Stitch Trigger on the sensors collection then runs a Stitch Function to push those sensor readings to AWS Kinesis for further analysis, such as anomaly detection.

The first appearance of the MongoDB rover coincided with the Mars landing of InSight. We even had our own 7 minutes of terror when the event WiFi buckled under the pressure of thousands of delegates descending on the expo hall during Monday’s evening social – but it did at least demonstrate that Stitch Mobile Sync quickly catches up once connectivity is restored.

This post is the first in a series delving into the components of this interactive demo, and if you can’t wait for more details, then you can find all of the code in GitHub.





Stitch & Mobile Webinar Questions & Replay

How do you test MongoDB Stitch functions, how do you store Stitch triggers, and what services can you integrate Stitch with? These were some of the great questions that were asked and answered in my recent webinar. You can watch the replay of “MongoDB Mobile and MongoDB Stitch – Introduction and Latest Developments” here, or read on as I share the answers to those questions here.

For those new to MongoDB Stitch, it’s the serverless platform from MongoDB that isolates complexity and ‘plumbing’ so you can build applications faster. Stitch went GA in June 2018, and we recently added a set of new capabilities, including global Stitch apps, more AWS services, a React Native SDK, and the beta for Stitch Mobile Sync. MongoDB Mobile is an embedded version of the MongoDB database that you can embed in your mobile or IoT apps.

Building mobile apps with MongoDB Mobile, MongoDB Stitch, and MongoDB Atlas

These are some of the questions I thought might be of interest:

How do you test Stitch functions?

The Stitch UI includes a console which can be used to test your Stitch functions; you can include console.log statements to add extra debug output to the console (they also get added to the log files for every function invoked from an incoming webhook, trigger, or the Stitch SDK).

You can also invoke your Stitch functions through the mongo shell. To do that, you’ll need to enable the MongoDB wire protocol so that the shell can talk to your Stitch app, then use the Stitch connection string provided. Once connected, you can call Stitch functions explicitly like this:

mongo> db.runCommand({callFunction: "morning", arguments: ["Billy"]})

{"ok" : 1,
 "response" : {"message" : "Good Morning Billy from andrew.morgan@mongodb.com"}
}

You can read more about this in this post which takes you through the process.

How do you store stitch triggers in your Git repo?

You can export your Stitch application from the Stitch UI or the Stitch CLI; the exported app is represented by a directory structure containing JSON and JavaScript files:

yourStitchApp/
├── stitch.json
├── secrets.json
├── variables.json
├── auth_providers/
│   └── <provider name>.json
├── functions/
│   └── <function name>/
│       ├── config.json
│       └── source.js
├── services/
│   └── <service name>/
│       ├── config.json
│       ├── incoming_webhooks/
│       │   ├── config.json
│       │   └── source.js
│       └── rules/
│           └── <rule name>.json
├── triggers/
│   └── <trigger name>/
│       ├── config.json
│       └── source.js
└── values/
    └── <value name>.json

You can then work on the trigger configuration and the associated function locally, source control the app using Git, or import it into a new App.

We saw a few integration provider logos in the presentation. Is there a page on the MongoDB site with the comprehensive list?

You can find the list of Stitch’s built-in service integrations in the Stitch documentation.

Note that if we don’t have a built-in integration for a particular service, then you can easily integrate it yourself, using the Stitch HTTP service and incoming webhooks. You can even export your new service integration to share with others.

Is MongoDB Mobile + Stitch Mobile Sync the same as a cache in a progressive app?

It certainly removes the need to have a separate cache, but it does much more. With MongoDB Mobile, the data is persistently stored on your device. You also have the full power of the MongoDB Query Language to perform sophisticated queries and aggregations on that local data. Changes made to the local database are pushed back to MongoDB Atlas, and from there to any other mobile devices configured to sync the same documents (e.g., for the same user running the app on another device).

How do I download & embed MongoDB Mobile?

You simply need to add 1 line to your Android or Xcode project to have access to the entire Stitch SDK, including the Stitch Local Database service (i.e.,the MongoDB Mobile database). The Stitch SDK includes the entire mobile and makes it very easy to use and consume, even if you’re just using the local MongoDB Mobile database and not Stitch.


Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Working with MongoDB Stitch Through Existing Drivers – Python & PyMongo

If you are developing an app using Java, Swift, or JavaScript then the Stitch SDK is the best way to access MongoDB Stitch from your frontend application code – getting to your data and accessing your Stitch Services and Functions becomes child’s play.

But, what if you are developing in another language – perhaps running on an app server? Luckily, MongoDB Stitch now supports the MongoDB wire protocol – meaning that you can continue to work with your favorite MongoDB drivers (such as PyMongo) and tools such as the mongo shell.

After enabling connection string access, connecting to your Stitch app from your application is business as usual – just use the connection string you’re shown in the Stitch UI (replacing the username and password with one of your existing Stitch users). This example is for Python, but the pattern is the same for other languages:

>>> import pymongo
>>> from pymongo import MongoClient
>>> client = MongoClient('mongodb://andrewxxxxmorgan%40gmail.com:\
      my_password@stitch.mongodb.com:27020/?authMechanism=PLAIN&\
      authSource=%24external&ssl=true&\
      appName=imported_trackme-etjzr:mongodb-atlas:local-userpass')

Inserting a document should seem very familiar (note that I’m including an owner_id attribute so that the app can take advantage of Stitch’s data access controls):

>>> db = client.trackme
>>> collection = db.comments

>>> comment = {
    "author" : "Andrew Morgan",
    "comment" : "Using Stitch from Python – who knew?!",
    "owner_id" : "5bacd4e7698a67f72dfdb44c"
}
>>> collection.insert_one(comment)

<pymongo.results.InsertOneResult object at 0x102b362d8>

Connecting to MongoDB Stitch using the MongoDB Python Driver

However, Stitch is about more just than accessing MongoDB data. I’ve created a (stupidly) simple Stitch (morning) Function to show how you can execute Stitch Functions through MongoDB drivers:

>>> db.command("callFunction", "morning", arguments=['Billy'])

u'ok': 1, u'response': {u'message': u'Good Morning Billy from andrewxxxxxxorgan@gmail.com'}}

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Working with MongoDB Stitch Through the mongo Shell – MongoDB Wire Protocol Support

The Stitch SDK is the best way to access MongoDB Stitch from your frontend application code – getting to your data and accessing your Stitch Services and Functions becomes child’s’ play.

However, those already using MongoDB may have existing backend code and tools that work with MongoDB. MongoDB Stitch now supports the MongoDB wire protocol – meaning that you can continue to work with your favorite tools (including the mongo shell) and drivers while benefiting from Stitch’s quick and simple data access control and hosted JavaScript functions.

After enabling connection string access, connecting to your Stitch app from the mongo shell is business as usual – just use the connection string you’re shown in the Stitch UI:

mongo "mongodb://andrewjamXXXXX%40gmail.com:gXXXX@stitch.mongodb.com:27020\
    /?authMechanism=PLAIN&authSource=%24external&ssl=true&\
    appName=imported_trackme-xxxxz:mongodb-atlas:local-userpass" --norc

Once connected, adding a document through Stitch should feel very familiar:

db.comments.insert({
    owner_id: "5bacd4e7698a67f72dfdb44c",
    author: "Andrew Morgan",
    comment: "Stitch wire protocol support rocks!"
})

MongoDB Stitch wire protocol

However, Stitch is about more than accessing MongoDB data. I’ve created a (stupidly) simple Stitch (morning) Function to show how you can test your Stitch app:

exports = function(name){
  return {message: "Good Morning " + name + " from " 
             + context.user.data.email};
};

From the mongo shell, I can now test that function:

db.runCommand({ callFunction: "morning", arguments: ["Billy"] })
{
    "ok" : 1,
    "response" : {
        "message" : "Good Morning Billy from greetings@clusterdb.com"
    }
}

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





MongoDB Stitch Authentication Triggers

MongoDB Stitch makes it a breeze to use third-party services such as Google or Facebook to authenticate your app’s users. This is great for your users as it means they don’t need to go through the tedious process of having to provide all of their details to yet another application/service – it also means one less password to remember.

While delegating the authentication process makes your app secure and straightforward to use, you may still want to customize the process for your users. For example, if someone is registering for the first time, then you probably want to send them a welcome email. When a user deregisters, there may be regulations (such as GDPR) that mandate you delete all of their data.

This is where Stitch Authentication Triggers come to the fore. The model is very simple; you register a Stitch Function with any authentication event (register (create), login, deregister (delete)), then add JavaScript code to that function to perform the required actions.

Simply create the trigger through the Stitch UI (in this case, to send the user an email every time they log in):

MongoDB Stitch Authentication Trigger

and implement the function (emailLoginNotification) that receives the authentication event and sends the email using the AWS SES service:

exports = function(event){
   var aws = context.services.get('AWS');
   var input = {
      Destination: {
         ToAddresses: [event.user.data.email]
      },
      Message: {
         Body: {
            Html: {
               Charset:"UTF-8",
               Data:   "This is a message to confirm that " 
                + event.user.data.name 
                + " (" + event.user.data.email
                + ") has just logged into TrackMe"
            }
         },
         Subject: {
               Charset:"UTF-8",
               Data:   "TrackMe Login"
         }
      },
      Source: context.values.get('fromEmail')
   };
   try {
      aws.ses().SendEmail(input).then(function (result){
         console.log(JSON.stringify(result));
      });
   } catch(error) {
      console.log(JSON.stringify(error));
   }
};

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Sending Text Messages with MongoDB Stitch & Twilio

When creating a well-rounded app, there are lots of table stakes features that make the app more useful but have already been implemented thousands of times before. Having the application backend send a text message informing your customer of an event is a classic example of such a “commodity” feature.

Think about using a website or app to book a taxi, where you give your phone number so that you get sent a text message when the taxi’s on its way. Why would the writers of that taxi app want to waste time writing text messaging code? There’s nothing extra they can do to differentiate it from other apps – so why not just consume it as a service from something like Twilio?

MongoDB Stitch makes it even less work to add this kind of feature. Rather than standing up an app server, figuring out how to use the Twilio API, writing the code, and possibly creating a REST API, just configure the Twilio service in Stitch.

Using MongoDB Stitch to send SMS text messages via Twilio

When configuring the Stitch Twilio serving, you supply the Account ID and Auth Token from your Twilio account. All that’s left is to write the Stitch function to invoke the service:

exports = function(trip, user){
  const twilio = context.services.get("twilio");

  twilio.send({
    to: user.phone,
    from: context.values.get("twilioNumber"),
    body: "Hi " 
      + user.firstname 
      + " - just to let you know that your taxi to " 
      + trip.destination + " will be with you at " 
      + trip.pickupTime + "."
  });
};

You can also have your application react when receiving a text message by configuring incoming webhooks for the Twilio service.

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Testing & Debugging MongoDB Stitch Functions

When discussing serverless computing (Functions as a Service) with developers, a common concern that arises is the complexity of testing and debugging your functions. Fortunately, the MongoDB Stitch UI makes this simple.

It’s a bit old school, but if you want to display debug info from your functions, then it’s as simple as adding console.log() commands to your code. If testing the function through the Stitch UI, the output appears in the Results panel. When executed normally, the output appears in the Stitch logs.

To test a Stitch function from the UI, select a user for the function to run as (that way the function can access whatever data the user is entitled to). In the Console panel, call exports(<parameters>), including any parameters that the function expects – these could be simple values or complex documents.

The results of the function call (the returned data + any console.log() output) appear in the Results panel.

Demo of testing a MongoDB Stitch Function

If you want to check on what’s happening in your production apps, check out the Logs panel in the Stitch UI.

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Building a REST API with MongoDB Stitch

One of the great things about MongoDB Stitch is that it often removes the need to build REST APIs to grant access to your data from frontend applications – simply use Stitch QueryAnywhere to make MongoDB queries from your frontend code. However, there are often cases where you need to open up some of your data to other applications which don’t use the Stitch SDK – fortunately, Stitch makes it incredibly easy to build REST APIs for these occasions.

I enjoy tracking my location by checking into Swarm/FourSquare, but I want to get some extra value from that data – that means getting it into a MongoDB collection.

FourSquare provides an IFTTT service that’s triggered whenever you check in – by linking that to the Maker service to send an HTTP POST request, I can forward that check-in data to Stitch. This is where Stitch comes in; a simple Stitch HTTP service webhook receives that POST request and writes the data to MongoDB:

exports = function(payload) {
  var queryArg = payload.query.arg || '';
  var body = {};
  if (payload.body) {
  body = EJSON.parse(payload.body.text());
  }

  var owner_id = context.functions.execute("ownerFromEmail", body.email);
  var checkin = {
    owner_id: owner_id.owner_id,
    email: body.email,
    venueName: body.venue,
    date: body.checkinDate,
    url: body.url,
    locationImg: body.location + "&key=" + context.values.get("GoogleMapsStaticKey")
  };

  return context.functions.execute("checkin", checkin);
};

The webhook uses the checkin function:

exports = function(checkin){
    var atlas = context.services.get("mongodb-atlas");
    var checkinColl = atlas.db("trackme").collection("checkins");
    try {
      checkinColl.insertOne(checkin);
    } catch (e) {
      console.log("Error inserting checkin doc: " + e);
      return e.message();
    }
};

Note that when configuring the HTTP service, I set an API key that the requestor must include as a secret query parameter:

Secure Stitch webhook with secret

IFTTT FourSquare applet

Now that the data is in MongoDB, there’s no limit to what I can do with it. For example, I want a dashboard for my check-in data, and one thing I want to include is a graph on my most frequent check-ins:

Foursquare check-in grapsh

To implement that, I write a new Stitch function:

exports = function(limit){
    var atlas = context.services.get("mongodb-atlas");
    var checkinColl = atlas.db("trackme").collection("checkins");
    var checkins = checkinColl.aggregate([
      {$match: {owner_id: context.user.id}},
      {$group: {
        _id: "$venueName",
        count: {$sum: 1}
      }},
      {$sort: {count: -1}},
      {$limit: limit},
      {$project: {
        venue: "$_id",
        _id: 0,
        count: 1
      }}
    ]).toArray();
    return checkins;
};

From the frontend application, it then takes a single method call to retrieve the data:

this.props.stitchClient
    .executeFunction('popularCheckins', 10)
    .then(
      checkinData => {
        this.setState({checkins: checkinData});
        this.createBarChart();
      },
      error => {
        console.log(
          "Failed to fetch most popular check-ins: "
          + error)
    })

You can recreate this Stitch and frontend app for your self by downloading the app from GitHub and importing it into Stitch.

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the white paper.





Recording sensor data with MongoDB Stitch & Electric Imp

Old school IoT

My early experiments with IoT involved standalone sensors, breakout boards, Arduinos, Raspberry Pis, and a soldering iron. It was a lot of fun, but it took ages to build even the simplest of projects.

Therefore, I was super excited when I discovered that Electric Imp had a self-contained IoT hardware developer kit and a library to access MongoDB Stitch directly.

Electric Imp impExplorer IoT development kit

My first experiment with Electric Imp took sensor data (temperature, humidity, air pressure, light level, and orientation) from the device, and stored it in MongoDB.

The Electric Imp code (written in Squirrel) is split into 2 parts:
– On-device code – sends sensor data to the agent
– Agent code, running in Electric Imp’s cloud – forwards the data to Stitch

This post focuses on the integration between the Electric Imp agent code and Stitch, but those interested can view the device code.

The agent code receives the readings from the device, and then it uses the Electric Imp MongoDB Stitch library to forward them to Stitch:

#require "MongoDBStitch.agent.lib.nut:1.0.0"

//Create the connection to Stitch
stitch <- MongoDBStitch("imptemp-sobpa");

//Add an API key to link this device to a specific Stitch User
const API_KEY = "hNErDmBw1zYGOfpaSv4Pf5kaNQrIaxOHLZgj0vExzDcxWf9GAEX055l1mXXX";

//Ensure you are authenticated to Stitch
stitch.loginWithApiKey(API_KEY);

function log(data) {
    stitch.executeFunction("Imp_Write", [data], function (error, response) {
        if (error) {
            server.log("error: " + error.details);
        } else {
            server.log("temperature sent");
        }
    });
}

// Register a function to receive sensor data from the device
device.on("reading.sent", log);

Note that you will need to create the API_KEY through the Stitch UI.

impExplorer sends readings to Electric Imp agent which sends them to MongoDB Stitch to store in MongoDB Atlas

The Imp_Write function receives the readings from the agent, retrieves outside weather data from DarkSky.net, and stores the data in the TempData MongoDB Atlas collection:

exports = function(data){

  //Get the current time
  var now = new Date();

  var darksky = context.services.get("darksky");
  var mongodb = context.services.get("mongodb-atlas");
  var TempData = mongodb.db("Imp").collection("TempData");

  // Fetch the current weather from darksky.net

  darksky.get({"url": "https://api.darksky.net/forecast/" + 
    context.values.get("DarkSkyKey") + '/' + 
    context.values.get("DeviceLocation") +
    "?exclude=minutely,hourly,daily,alerts,flags&units=auto"
  }).then(response => {
    var darkskyJSON = EJSON.parse(response.body.text()).currently;

    var status =
      {
        "Timestamp": now.getTime(),
        "Date": now,
        "Readings": data,
        "External": darkskyJSON,
      };
    status.Readings.light = (100*(data.light/65536));
    context.functions.execute("controlHumidity", data.temp, data.humid);
    TempData.insertOne(status).then(
      results => {
        console.log("Successfully wrote document to TempData");
      },
      error => {
        console.log("Error writing to TempData collection: " + error);
      });
  });
};

ImpWrite also calls the controlHumidity method – find more on that in this post.

You can recreate the Stitch app for yourself by downloading the app from GitHub and importing it into Stitch. You’ll need to set some of your own keys first (including the details of your IFTTT webhook address) – details are in the README. The repo also includes the Electric Imp code for the agent and device.

Creating your first Stitch app? Start with one of the Stitch tutorials.

Want to learn more about MongoDB Stitch? Read the MongoDB Stitch white paper.