Tracking Dependencies With MongoDB 4.0

Rich DammkoehlerMon, 03/18/2019 - 09:44

I've been a big fan of MongoDB since I was first introduced to in back in 2011. I've used it for a slew of projects. In 2011, we used MongoDB as a transaction log. Elsewhere, I've used it to track software packaging and distribution. We used it as a global log file on one project, and last year, I started on a project using MongoDB as an encrypted log database, tracking every piece of data sent back to the UI.

MongoDB is so easy to use and integrates with so many things I can't stop finding ways to bring it into my projects.

I recently started playing with MongoDB again, using MongoDB 4.0 in a Docker container just to do some experiments. I have a project where I want to capture build data for each distribution. The software is a containerized multi-service application with hundreds of independent dependencies. For legal reasons, we need to track every single dependency in a distribution package. So I'm using MongoDB to capture this information.

In this article, I'll show you how I'm setting up MongoDB 4.0 in a Docker container. Then, using the command line client, I'll build a prototype of the data model I have in mind. Finally, I'll demonstrate a technique for updating multiple documents and subdocuments using the forEach() function. After seeing how easy the setup is, maybe you'll want to use MongoDB in your own project!

Getting MongoDB 4.0

Let's start this experiment by getting MongoDB 4.0 running in Docker. It's trivial to get a container running with this command:

tracking-dependencies-with-mongoDB-tutorial-1

As you would expect, Docker dutifully downloaded the MongoDB 4.0 image from Docker hub and started up a container named mongo_sandbox.

Starting Mongo Shell

In order to execute commands on the running MongoDB, we need the Mongo Shell. Since that comes with the server, we're just going to open a shell on the running server. In order to do that, we need to connect to the running Docker container using the following command:

tracking-dependencies-with-mongoDB-tutorial-2

You should see your prompt change from its default to the pound sign (#). You now have a shell on the MongoDB server. At the shell prompt, just type the mongo command:

tracking-dependencies-with-mongoDB-tutorial-3tracking-dependencies-with-mongoDB-tutorial-4

You'll get the typical startup messages warning you not to use the default XFS file system and telling you that access control isn't configured. For our experiment, this is fine. But I know from previous experience that these warnings are real. We would follow the recommended approach if we were doing something more serious. For this spike, let's not bother.

Exploring MongoDB 4.0

MongoDB 4.0 starts up with three databases, to begin with. You can see these by using the show dbs command in the Mongo Shell.

tracking-dependencies-with-mongoDB-tutorial-5

The three databases are admin, config, and local.

Create Our Prototype DB

Creating a database in MongoDB is easy, but it's unusual. Rather than run something you'd expect, like create <db>, the command is use <db>, where <db> is the name of the database you want to use. If it doesn't exist, MongoDB creates it for you.

tracking-dependencies-with-mongoDB-tutorial-6

After that, we'll create the database collection with the createCollection command.

tracking-dependencies-with-mongoDB-tutorial-7

Let's verify that our collection is there:

tracking-dependencies-with-mongoDB-tutorial-8

It is!

Data Structure

For our prototype, let's use a document structure like this:

tracking-dependencies-with-mongoDB-tutorial-9

The idea here is that, for any deployment, we can create a document that captures information about what was deployed and when. For our first experiments, this is just fine; when it comes to the real thing, things will be more complicated.

As you can see, our document structure includes key-value pairs for version, deployment code name, packaging date, and the like. There's also an array for storing tags. The components array is supposed to represent an array of deployed components, each of which has a structure. These are called subdocuments. This is all just your run-of-the-mill JSON file in the end.

Inserting Data

Let's insert some documents into our collection. First, create an abbreviation for our collection. This just shortens up what we type.

tracking-dependencies-with-mongoDB-tutorial-10

Then we can insert our first document:

tracking-dependencies-with-mongoDB-tutorial-11 https://gist.github.com/rdammkoehler/cbc4cee64fe79b28ffac2475900901c3

You can see that the server responds with a simple acknowledgment that includes the document's ObjectId.

tracking-dependencies-with-mongoDB-tutorial-12

Just to prove the document is really there, I used a simple find() command to get all the documents in the collection:

tracking-dependencies-with-mongoDB-tutorial-13

Let's add one more record to the collection:

https://gist.github.com/rdammkoehler/be3ef612fa82043c0d3d6248b496132a

Bulk Updates

Notice in the output that our packaging_date is actually a string. What we really want to do is store that as an ISODate. If we had a lot of records, updating would be a tedious project. Fortunately, MongoDB makes doing a general update pretty easy.

An update cannot refer to the record being updated. This makes some kind of rational sense because you're changing the structure of the document and the query evaluation could become hopelessly tangled. So, you need to use the forEach() operation on a query result to perform the update.

First, let's fix the packaging_date field with the following command:

tracking-dependencies-with-mongoDB-tutorial-14

This code uses find with no arguments to bring back all the records in our collection. The hint() method overrides the default indexing of the query. For our operation, this isn't strictly necessary. But I've included it here just to demonstrate that it works. The forEach operation accepts a function that's applied to each element in the result set. So e represents a record in the result.

Our function simply updates the packaging_date field with a new Date instance initialized with the string version we entered above. This is followed by coll.save(e), which is telling the collection to save the record we just created. Here, we've used the abbreviated collection name coll that we created back in the beginning of the exercise. We could just as easily use db.deployments, but coll is shorter.

Now when we look at the data using the simple find() command, we can see that the packaging_date field is stored as an ISODate rather than a string.

tracking-dependencies-with-mongoDB-tutorial-15

Pretty Printing

You might be finding it hard to read the output. There's a pretty printer built in that you can use like this:

tracking-dependencies-with-mongoDB-tutorial-16

Processing the Subdocuments

In order to process the subdocuments, we need to loop through both the documents in our collection and the array of components within those documents. Not surprisingly, we can do this with another forEach inside of the one we already created.

The following code will update the subdocuments component build_date from a string to an ISODate:

tracking-dependencies-with-mongoDB-tutorial-17

So this approach is similar to the previous one, but we're going to iterate through all of the components listed in the array that are named components in our document structure. This is just another forEach that contains a function. The end result is that each build_date is converted from a string to an ISODate.

And when we print them again, we see that each of the build_date fields was correctly converted:

tracking-dependencies-with-mongoDB-tutorial-18

Conclusion

I'm really excited about playing with MongoDB again. Some things have changed in MongoDB 4.0, but much of the power and simplicity still exists. In particular, I'm looking forward to the new features in MongoDB 4.0, like multi-document ACID transactions, the changes to the internal security, and the additional functions I see listed in the release notes.

What are you excited to use MongoDB for? Hopefully, this gave you some ideas of how it could be useful in your own projects!

Looking for a more in-depth look at MongoDB's capabilities, check out our course MongoDB Development Boot Camp. In this MongoDB training course, you’ll gain a working knowledge of the full capabilities of the tool, including how to model documents and create relationships between those documents. You’ll gain a greater understanding of queries and sorting. And you’ll improve your queries with detailed explanations of indexing techniques.