Mongod

This post will serve you as notes for the mongod section of the MongoDB M103 Basic Cluster Administration Course

Definitions

• Daemon : A daemon is a program or process that is running but is not directly interacted with.
• Mongod : mongod is the main daemon process for mongodb

Mongod

• Default Configuration :
  • Default Port : 27017
    • To change the default port with the new one, use the following command :

      			mongod --port <port number>
      	

      			mongod --port <port number>
      	

  • Default DB path : /data/db
    • This is the path where all the files and directories related to databases, indexes are stored.
    • This is the path where the journaling information is stored so that the data remains consistent even after an unexpected crash.
    • To change the default db path with the new one use the following command :

      			mongod --dbpath <directory path>
      	

      			mongod --dbpath <directory path>
      	

  • Default bind_ip : localhost
    • By default mongod process binds to localhost
    • This means that only the clients that are installed in the local machine will be able to connect to the database or mongod daemon.
    • To change the default bind_ip use the following command :

      			mongod --bind_ip 123.123.123.123
      	

      			mongod --bind_ip 123.123.123.123
      	

    • To bind the mongod process with multiple ip use the following command :

      			mongod --bind_ip localhost,123.123.123.123
      	

      			mongod --bind_ip localhost,123.123.123.123
      	

  • Default Authentication :
    • auth : disabled
    • This means that unless we enable the authentication the clients connecting to the database do not need to authenticate before accessing the data.
    • To enable authentication in mongodb :

      			mongod --auth
      	

      			mongod --auth
      	

• Some default information of the mongod is always displayed in the logs when starting the mongod process.
• Every command displays a log message in the mongod logs
• To shutdown the mongodb server :
  • Switch to admin database using the command :

    			use admin
    	

    			use admin
    	

  • Issue the following command to shut down the server

    			db.shutdownServer()
    	

    			db.shutdownServer()
    	

  • To exit out of the mongodb shell use the following command :

    			exit
    	

    			exit
    	

• Other clients to connect to mongodb shell :
  • MongoDB Compass
  • Drivers of the particular language :
    • Node, Swift, Java, C/C++. etc…
• Options available with mongod :
  1. —dbpath
  2. —logpath
  3. —bind_ip
  4. —replSet
  5. —keyFile
  6. —sslPEMKey
  7. —sslCAKey
  8. —sslMode
  9. —fork

Configuration File

• Configuration file in mongodb provides an easy way to change the default options that are specified with the mongod command as arguments.
• We can also use comments in the configuration file which improves readability.
• Configuration file is written in YAML which refers to YAML Ain't Markup Language
• Counterparts for the above mongod options in the configuration file
  1. storage.dbPath
  2. systemLog.Path and systemLog.destination
  3. net.bind_ip
  4. replication.replSetName
  5. security.keyFile
  6. net.ssl.sslPEMKey
  7. net.ssl.sslCAKey
  8. net.sslMode
  9. processManagement.fork
• To understand the differences between passing the arguments to mongod process and using a configuration file have a look at the below code :
  • Providing arguments to mongod process

    			mongod --dbpath /data/db 
    --logpath /data/log/mongod.log 
    --fork --replSet "M103" 
    --keyFile /data/keyfile 
    --bind_ip "127.0.0.1,192.168.103.100" 
    --tlsMode requireTLS 
    --tlsCAFile "/etc/tls/TLSCA.pem" 
    --tlsCertificateKeyFile "/etc/tls/tls.pem"
    	

    			mongod --dbpath /data/db 
    --logpath /data/log/mongod.log 
    --fork --replSet "M103" 
    --keyFile /data/keyfile 
    --bind_ip "127.0.0.1,192.168.103.100" 
    --tlsMode requireTLS 
    --tlsCAFile "/etc/tls/TLSCA.pem" 
    --tlsCertificateKeyFile "/etc/tls/tls.pem"
    	

  • Using a configuration file written in YAML

    			storage:
      dbPath: '/data/db'
    systemLog:
      path: '/data/log/mongod.log'
      destination: 'file'
    replication:
      replSetName: M103
    net:
      bindIp: '127.0.0.1,192.168.103.100'
    tls:
      mode: 'requireTLS'
      certificateKeyFile: '/etc/tls/tls.pem'
      CAFile: '/etc/tls/TLSCA.pem'
    security:
      keyFile: '/data/keyfile'
    processManagement:
      fork: true
    	

    			storage:
      dbPath: '/data/db'
    systemLog:
      path: '/data/log/mongod.log'
      destination: 'file'
    replication:
      replSetName: M103
    net:
      bindIp: '127.0.0.1,192.168.103.100'
    tls:
      mode: 'requireTLS'
      certificateKeyFile: '/etc/tls/tls.pem'
      CAFile: '/etc/tls/TLSCA.pem'
    security:
      keyFile: '/data/keyfile'
    processManagement:
      fork: true
    	

File Structure

These are the typical files present in the /data/db directory

• WiredTiger.lock :
  • WiredTiger prevents multiple connection handles by writing a lock file, and this locking is done even in read-only mode.
• Files ending with .wt :
  • There are two types of files that end with .wt
    • One is the files related to collections
    • Another one is the files related to indexes
  • Each collection is assigned a file that ends with .wt
• diagnostic.data :
  • This directory contains the diagnostic data collected to be analysed by the mongodb support engineers.
  • These files can only be viewed by the mongodb support engineers only if we explicitly give permission to see the files.
• journal :
  • This directory contains the logs which is generated by each process initiated like write,read,modify
  • The files in this directory are used to recover from the crash using the below process :
    1. Looks in the data files to find the identifier of the last checkpoint.
    2. Searches in the journal files for the record that matches the identifier of the last checkpoint.
    3. Apply the operations in the journal files since the last checkpoint.
• mongodb-27017.sock :
  • This is a sockets file used by the mongod process to establish connections using sockets.
  • This file ensures that mongod process owns the port.
  • It is safe to delete this file in the event of a crash and failure to start the mongod process.

Basic Commands

• User Management commands :
  • db.createUser()
  • db.dropUser()
• Collection Management commands :
  • db.renameCollection()
  • db.collection.createIndex()
  • db.collection.drop()
• Database Management commands :
  • db.dropDatabase()
  • db.createCollection()
• Database Server status commands :
  • db.serverStatus()
• Difference between database commands and shell helpers :
  • Create index using database command :

    			db.runCommand({
      createIndexes: '<collection_name>',
      indexes: [
        {
          key: { product: 1 },
          name: 'name_index',
        },
      ],
    })
    	

    			db.runCommand({
      createIndexes: '<collection_name>',
      indexes: [
        {
          key: { product: 1 },
          name: 'name_index',
        },
      ],
    })
    	

  • Create index with shell helper :

    			db.<collection>.createIndex(
      { "product": 1 },
      { "name": "name_index" }
    )
    	

    			db.<collection>.createIndex(
      { "product": 1 },
      { "name": "name_index" }
    )
    	

• Introspect a shell helper :

  			db.<collection>.createIndex
  	

  			db.<collection>.createIndex
  	

Logging Basics

Information showed in the mongod process log :

• ACCESS : authentication

• COMMAND : database commands

• CONTROL : initialization

• FTDC : diagnostic data collection mechanism

• GEO : parsing of geospatial objects

• INDEX : indexing operations

• NETWORK : accepting connections

• QUERY : query planner activities

• REPL : related to replica set such as initial sync or heartbeats

• REPL_HB : replica set heartbeats

• ROLLBACK : rollback operations

• SHARDING : sharding operations

• STORAGE : storage activities

• JOURNAL : journaling activities

• WRITE : update commands

• To know the information about the verbosity level for each of the section displayed in the mongod logs use the following command :

  			db.LogComponents()
  	

  			db.LogComponents()
  	

• The verbosity levels mentioned in the output of the above command :

  • -1 : Inherit from parent
  • 0 : Default verbosity. to include informational messages
  • 1 - 5 : increases the verbosity level to include Debug messages

• To get the logs using the mongo shell :

  			db.adminCommand({ getLog: 'global' })
  	

  			db.adminCommand({ getLog: 'global' })
  	

• Another method to view the logs is using the below command :

  			tail -f /data/db/mongod.log
  	

  			tail -f /data/db/mongod.log
  	

• To set the verbosity level for a particular section, for example index use the below command :

  			db.setLogLevel(0, 'index')
  	

  			db.setLogLevel(0, 'index')
  	

• Information present in the debug message in the mongod logs :

  • Lorem ipsum

Profiling the Database

• Need for the profiler in the mongod process :
  • Though many of the information are displayed in the logs of the mongod process, there are no information regarding execution stats, the direction of an index used by a query, rejected plans etc…
  • There is a way to add the above info to the mongod logs but it should not be done. Because of the fact that the logs provide the administrators with the operational logs about an instance or process so that they can find any errors, warnings, or any interesting informational messages.
• Profilers are enabled on database level, so each database will have a profile.
• All the information regarding the profiling is stored in a seperate collection named system.profile
• When the profiler is enabled it collects the data about CRUD operations, Administrative operations and Configuration options.
• The profiler has three settings :
  • 0 : The profiler is off and does not collect any data. This is the default profiler level
  • 1 : The profiler collects data for the operations that take longer than the value of slowms.
    • By default any operations that take longer than 100ms is considered slow by mongod.
  • 2 : The profiler collects data for all operations.
    • This profiler setting is considered dangerous, since it will log all operations that can generate huge amounts of write to the system.profile collection and put a load on the system.
• To get the status about profiling level on the active database use the following command :

  			db.getProfilingLevel()
  	

  			db.getProfilingLevel()
  	

• To change the value of the profiling level on the active database use the following command :

  			db.setProfilingLevel(1)
  	

  			db.setProfilingLevel(1)
  	

• To set the slowms value use the following command :

  			db.setProfilingLevel(1, { slowms: 0 })
  	

  			db.setProfilingLevel(1, { slowms: 0 })
  	

• To get the data collected in the system.profile collection use the following command :

  			db.system.profile.find().pretty()
  	

  			db.system.profile.find().pretty()
  	

Basic Security : Part 1

Authentication

Types of Authentication Mechanisms :

• Community and Enterprise Versions :
  • SCRAM : Salted Challenge Response Authentication Mechanism
    • Basically password authentication
  • X.509 : Certificates
• Enterprise Versions only :
  • LDAP : Lightweight Directory Access Protocol
    • LDAP is a way of organizing and storing information about people, groups, and computers in a central place. Think of it as a digital address book that can be accessed from anywhere on a network.
  • KERBEROS :
    • Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography

Authorization

Role based access control :

• Each user has one or more roles
• Each role has one or more privileges
  • A privilege represents a group of Actions and Resources those actions apply to.

Basic Security : Part 2

Localhost Exception

• Allows you to access Mongodb server that has authentication enabled but does not yet have a configured user for you to authenticate with.
• Must run mongo shell from the same host running the mongodb server
• The localhost exception closes after the first user is created.
• Always the first user created should have administrative privileges.

Create a administrative user

The administrative user can be created with the localhost exception

To create the administrative user use the following command in the mongo shell :

			db.createUser({
  user: 'Username',
  pwd: 'Password',
  roles: [{ role: 'root', db: 'admin' }],
})
	

			db.createUser({
  user: 'Username',
  pwd: 'Password',
  roles: [{ role: 'root', db: 'admin' }],
})
	

To connect to the mongod instance using the client authentication method use the following command :

			mongo --host 127.0.0.1 
--username root 
--password root 
--authenticationDatabase admin
	

			mongo --host 127.0.0.1 
--username root 
--password root 
--authenticationDatabase admin
	

Builtin Roles

Role Structure :

• Role is composed of
  • Set of privileges
    • Actions → Resources
• Roles can be also inherited
• Network level restrictions can also be imposed on the roles meaning :
  • Roles can connect to a resource from a clientSource
  • Roles can connect to a serverAddress
  • Network restrictions can be implemented by specifying the clientSource and serverAddress

Resources :

Resources can be of many types :

• Specific database and collection
• All databases and all collections
• Any database and specific collection
• Specific database and any collection
• Cluster resource

Privilege :

• Specifies the resource to act on
• Actions allowed over the resource
• For example : Allow to shutdown the cluster

  			{ "resource": { "cluster": true }, "actions": ["shutdown"] }
  	

  			{ "resource": { "cluster": true }, "actions": ["shutdown"] }
  	

Builtin Roles :

There are 5 types of builtin roles which have specific privileges prepackaged in the mongod

• Database user :
  • read
  • readWrite
• Database Administration :
  • dbAdmin
  • userAdmin
  • dbOwner
• Cluster Administration :
  • clusterAdmin
  • clusterManager
  • clusterMonitor
  • hostManager
• Backup / Restore :
  • backup
  • restore
• Super User :
  • root

There are particular roles in the builtin roles that have access to any database :

• Database user :
  • readAnyDatabase
  • readWriteAnyDatabase
• Database Administration :
  • dbAdminAnyDatabase
  • userAdminAnyDatabase
• Super User :
  • root

userAdmin :

Privileges granted to users with userAdmin role :

• changeCustomData
• changePassword
• createRole
• createUser
• dropRole
• dropUser
• grantRole
• revokeRole
• setAuthenticationRestriction
• viewRole
• viewUser

dbAdmin :

Privileges granted to users with dbAdmin role :

• collStats
• dbHash
• dbStats
• killCursors
• listIndexes
• listCollections
• bypassDocumentValidation
• collMod
• collStats
• compact
• convertToCapped

dbOwner :

• The database owner can perform any administrative actions on the database
• This role combines the privileges granted by readWrite, dbAdmin, userAdmin roles

To grant a role to a user use the following command :

			db.grantRolesToUser( "user",  [ { db: "database", role: "role"  } ] )
	

			db.grantRolesToUser( "user",  [ { db: "database", role: "role"  } ] )
	

To view the information about the role privileges use the following command :

			db.runCommand( { rolesInfo: { role: "role", db: "database" }, showPrivileges: true} )
	

			db.runCommand( { rolesInfo: { role: "role", db: "database" }, showPrivileges: true} )
	

Server Tools Overview :

There are many tools that come along when we install mongodb

The list of the tools that we need are :

• mongostat :
  • The output of the mongostat command lists the following details :
    • Insert, query, update, delete, getmore, command
      • These fields represent the no. of operations occuring per second
    • dirty :
      • percentage of dirty bytes in the cache
    • used :
      • percentage of bytes used in the cache
    • vsize :
      • amount of virtual memory used by the process
    • res :
      • amount of resident memory used by the process
    • net_in :
      • amount of network traffic that is recieved
    • net_out :
      • amount of network traffic that is sent
• mongoimport :
  • This is the inverse of the mongoexport command
  • Since we do not provide a database and collection name while using this command it defaults to test as the database and name of the collections is picked up from the name of the file.
• mongoexport :
  • This command deals with outputting the information or the collections present in the mongodb in json format instead of bson
  • This command does not create a dump directory
  • Example command :

    			mongoexport --port 30000 --db applicationData --collection products -o products.json
    	

    			mongoexport --port 30000 --db applicationData --collection products -o products.json
    	

• mongodump :
  • used to dump the collections in the mongodb
  • When the command is used with several arguments such as username and password it creates a dump directory which contains the directory named by the collection you dumped.
    • The directory contains a .bson file which contains the actual information and metadata.json file which contains the information about the indexes in the collections
  • Example of the command :

    			mongodump --port 30000 --db applicationData --collection products
    	

    			mongodump --port 30000 --db applicationData --collection products
    	

• mongorestore :
  • This command is the inverse of the mongodump command which takes the .bson file and creates a database collection from it.
  • Example command which drops the current collection present and replaces with the collection created by the .bson file

    			mongorestore --drop --port 30000 dump/
    	

    			mongorestore --drop --port 30000 dump/