lessons learned using mongodb

Back in 2012, there was a new way to build web apps fast and packed full of features. It was MEAN. MongoDB, Express, Angular, and Node. I built a number of apps with this technology, and it worked. However, as time passed they began to show their weaknesses. Today we’ll look at a few lessons learned from MongoDB.

What MongoDB Kicks Butt At

Most technologies cannot be just dismissed outright, even though you read article after article telling you “Why you should not X”. That’s just stupid, narrow minded, and immature.

You should use X for what X was best designed to do.

MongoDB is a Document Store, not to be confused with a database as you are probably thinking. Relational Databases have been around since the dawn of unix timestamp. MongoDB is really good for two types of data.

• Data that does not change much
• Data with different nested structure you don’t really care about consistent fields

For example. Let’s say you have a device and it logs a large number of things about itself every second. Say, a phone that logs its latitude, longitude, battery charge, network information, and more. Later you’ll want to query or search this information, but it doesn’t need to be updated since you are recording a log. MongoDB would be great for this. Since it can handle huge amounts of data and it really doesn’t have to be consistent.

What MongoDB doesn’t do well

The appeal to me initially was being able to keep the schema of my data in the app and have the app worry about field names. MongoDB doesn’t care if you feed it different types of data with each insert into a collection. It will happily create the corresponding field in a related data format. It might not be the type of data you want, but life goes on.

Missing or misspelled fields

While migrating the data I realized that one of the timestamp fields was named creatdAt. Not a big deal since I wasn’t really using this field on the front end at the time, but just think if I would have realized this in the code and updated it? Then I would have a boat load of records with a creatdAt field, and another set of the same records with a createdAt field. You now have to write a script to fix all the incorrect spellings.

Fields with different data types

While migrating the data I had number fields that were supposed be numbers, but in fact some were strings, some floats, some integers. I had to adapt my import script to look for different types of data (string, number, float) and then convert them into integers. Notice the $numberLong, that’s sometimes when the integer is too long for Javascript, so it will use that type of indication. numberLong.

function getAmount(amount) {
  if (amount === undefined) {
    return 0
  }

  if (isNumber(amount)) {
    return parseInt(amount, 10)
  }

  if (isString(amount)) {
    return parseInt(amount, 10)
  }

  if (amount && amount['$numberLong']) {
    return parseInt(amount['$numberLong'], 10)
  }

  return 0
}

Inconsistency in your data

For some weird reason, a few of the passwords in the database were being stored as plain text instead of a hash. This was very disconcerting. I’ll write more later on why you should not be doing password hashing in the app (hint: let the database do it). In other cases, since I didn’t do a lot of data validation on some fields, I noticed that people were using the app in very strange ways. Like storing weird meta values in a person’s email record. I guess they were just being creative in how they stored data.

Conclusion

Building a web app with users, data, settings, etc? Use Postgres as your database. If you need to store massive amounts of log or complex JSON object data, then use MongoDB. As an added bonus, Postgres has so many built in features, it can actually do many of the things you were doing in the app. Such as updating timestamps, computing user password hashes, and much more.

But What About Migrations of Schema

Since I use Node.js mostly, I use the wonderful Knex module that you can write migrations in this manner

const TABLE = `charges`

exports.up = function (knex, Promise) {
  return knex.schema.alterTable(TABLE, t => {
    t.jsonb('shipping_contact').comment('JSON object of shipping details')
  })
}

exports.down = function (knex, Promise) {
  return knex.schema.alterTable(TABLE, t => {
    t.dropColumn('shipping_contact')
  })
}

I have an npm script that looks like this "start": "npm run migrate && node ./web"

Did this post help? I’d love to hear from you.