paging – Adding multiple indexes to avoid skip/limit in MongoDB

Let’s say I have a MongoDB’s database with millon of records having a City name, a User name and an activity.

City        | User   | Action
------------+--------+-------------------
Gotham      | Batman | Fight with Joker
Gotham      | Batman | Fight with Penguin
Gotham      | Robin  | Fight with Joker
Gotham      | Robin  | Fight with Riddler
Springfield | Homer  | Working
Springfield | Homer  | Moe's bar
Springfield | Bart   | At school

I want a fast paging method to show data in a web site listing all results, results by city or results by city and user. Although I have the main indexes by city & name, still using the db.find(...).skip((page_number - 1) * rows_per_page).limit(rows_per_page) method is slow.

Because no record is deleted, my approch to circumvent the performance issue, is to add some extra rows with incremental values and several new indexes:

City        | User   | Action             | Global Index | Index By City | Index By City & User
------------+--------+--------------------+--------------+---------------+---------------------
Gotham      | Batman | Fight with Joker   | 1            | 1             | 1
Gotham      | Batman | Fight with Penguin | 2            | 2             | 2
Gotham      | Robin  | Fight with Joker   | 3            | 3             | 1
Gotham      | Robin  | Fight with Riddler | 4            | 4             | 2
Springfield | Homer  | Working            | 5            | 1             | 1
Springfield | Homer  | Moe's bar          | 6            | 2             | 2
Springfield | Bart   | At school          | 7            | 3             | 1

So I do paging by Global Index, City + Index By City or City + User + Index By City & User

Is this approach good or horrible? Or is there a better method?

mongoose – Documentos embebidos o referencia, mongoDB

Tengo una duda puntual en este tema, estoy practicando y quiero hacer un clon de Trello. El tema me surge en el modelado de la base de datos. En la parte de tablero pensaba hacer las listas de tareas como documentos embebidos y su vez las tareas embebidas en la listas. Yo razono así por el hecho de que estas seran limitadas y solo seran usadas dentro de su proyecto padre. Esta bien esto o puedo causarme algun problema soy nuevo con el NoSql, o me conviene hacerla por referencias en un documento aparte como sql y porque?

backup – how to migrate a mongodb over a couple of versions safely & efficient?

what is the best method to migrate a database (or at least 1 of 6 collections) from mongodb 2.4 to 4.4 with “reasonable effort”?

  • i know there is mongodump/mongorestore, which is considered the “more accurate” method and is advised for backup & restore on the one hand…
    but as far as i’ve read, the mongodb versions must be the same and they can only be upgraded one after another! so i would have to upgrade 8 times (2.4 -> 2.6 -> 3.0 -> 3.2 -> 3.4 -> 3.6 -> 4.0 -> 4.2 -> 4.4) manually, which is pretty tedious; not to mention, that not all versions are available for the old machine (Debian 8 with SysVinit).
    btw: i tried to install mongodb 4.4 on Debian 10 with SysVinit, which failed because of systemd missing (no, not just starting the service, indeed the installation)! 🙁
    –> if upgrading to every version takes as much fiddling around & time as installing mongodb 4.4 on Debian 10 with SysVinit did, the whole process would take up to a week, which is insane!

  • but on the other hand there is mongoexport/mongoimport which is more or less only the tool of choice, if a 3rd party is involved, because it can cause data loss.
    is it nevertheless worth a try in this situation? how can i check for a data loss resp. avoid or even better be sure, that there is no data loss? is using mongoexport --jsonFormat=canonical (...) & mongoimport --stopOnError (...) and then comparing the number of exported & imported records sufficient, to be on the safe side?

or is even using a dump from mongodb 2.4 to restore at mongodb 4.4 worth a try, despite all docs and tutorials? how can i check easily for data loss resp. messed up data in this case?

to mention it, just in case: i’m kind of an advanced linux user, but by no means a database expert! so please bear with me, if i’ve got something wrong.

How to populate array from mongoDb un Dotnet Core?

I would like to inquire regarding the query of data from a database in MongoDb.

I’m learning about it, and the documentation is still a bit difficult for me to understand.

The thing is, I want to fetch some data from a collection that contains an array of IDs from another collection.

In Javascript the ** populate ** instruction would be used, but I still don’t understand how I can use ** linq ** or ** Lookup ** to get the same result.

The entities are basically:

public sealed class Person
    {
        (BsonId)
        (BsonRepresentation(BsonType.ObjectId))
        public String id { get; set; }
        public List<Qr> codigoQr { get; set; } = new List<Qr>();
    }

and

public sealed class Qr
    {
        public Qr()
        {}

        (BsonId)
        (BsonRepresentation(MongoDB.Bson.BsonType.ObjectId))
        public String id { get; set; }
    }

To get the people from the Db I am using:

var docs = _persons.Aggregate()
                     .Lookup("Qr", "id", "codigoQr", "asAccounts")
                     .As<MongoDB.Bson.BsonArray>()
                     .ToList();

            return docs;

but i get the error:

System.FormatException: An error occurred while deserializing the codigoQr property of class HCS.Models.Person: Expected a nested document representing the serialized form of a HCS.Models.Qr value, but found a value of type ObjectId instead.

I have tried ** Linq ** but I don’t understand how to use the ** join ** statement to populate the data in the array.

Basically I have in db a list of people with the following format:

{
    "_id": {
        "$oid": "5f5993182939e4f2446ed47a"
    },
    "name": "Pablo",
    "lastname": "Mederos",
    "codigoQr": ({
        "$oid": "5f59948f2939e4f2446ed47c"
    },
    {
        "$oid": "5f59948f2939e4f2446ed47c"
    })
}

Is there a way to populate that list with the data from the Qr collection?

From already thank you very much!

.net – C# MongoDb Driver, Cómo debería poblar un array?

Quisiera consultar referente a la consulta de datos de una base de datos en MongoDb.

Estoy aprendiendo al respecto, y la documentación se me hace un poco difícil de entender aún.

El asunto es que quiero traer algunos datos de una colección que contiene un array de Ids de otra colección.

En Javascript se utilizaría la instrucción populate, pero no entiendo aún como puedo utilizar linq o Lookup para obtener el mismo resultado.

Las entidades básicamente son:

public sealed class Person
    {
        (BsonId)
        (BsonRepresentation(BsonType.ObjectId))
        public String id { get; set; }
        public List<Qr> codigoQr { get; set; } = new List<Qr>();
    }

y

public sealed class Qr
    {
        public Qr()
        {}

        (BsonId)
        (BsonRepresentation(MongoDB.Bson.BsonType.ObjectId))
        public String id { get; set; }
    }

Para obtener las personas de la Db estoy usando:

var docs = _persons.Aggregate()
                     .Lookup("Qr", "id", "codigoQr", "asAccounts")
                     .As<MongoDB.Bson.BsonArray>()
                     .ToList();

            return docs;

pero obtengo el error

System.FormatException: An error occurred while deserializing the codigoQr property of class HCS.Models.Person: Expected a nested document representing the serialized form of a HCS.Models.Qr value, but found a value of type ObjectId instead.

He intentado con Linq pero no entiendo como utilzar la instrucción join para poblar los datos en el array.

Básicamente tengo en db una lista de personas con el siguiente formato:

{
    "_id": {
        "$oid": "5f5993182939e4f2446ed47a"
    },
    "name": "Pablo",
    "lastname": "Mederos",
    "codigoQr": ({
        "$oid": "5f59948f2939e4f2446ed47c"
    },
    {
        "$oid": "5f59948f2939e4f2446ed47c"
    })
}

Existe alguna forma de poblar esa lista con los datos de la colección Qr?

Desde ya muchas gracias!

mongodb – Mongo Database – Dirty Shutdown

We have a major issues on our MongoDB and I’ve attached the latest logs and files. In previous forumns I’ve seen this get fixed by your team. Can you please help as soon as possible please, we are a medical facility and need this fixed.

We are running on Windows 2016 Server and running fine until a power outage.

2020-09-11T14:34:18.780-0400 I CONTROL (main) Trying to start Windows service ‘MongoDB’
2020-09-11T14:34:18.781-0400 I CONTROL (initandlisten) MongoDB starting : pid=1816 port=27017 dbpath=C:Program FilesDELLWMSDatabaseMongo 64-bit host=VLDATAWMS02
2020-09-11T14:34:18.781-0400 I CONTROL (initandlisten) targetMinOS: Windows Vista/Windows Server 2008
2020-09-11T14:34:18.781-0400 I CONTROL (initandlisten) db version v3.4.10
2020-09-11T14:34:18.781-0400 I CONTROL (initandlisten) git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2020-09-11T14:34:18.781-0400 I CONTROL (initandlisten) allocator: tcmalloc
2020-09-11T14:34:18.782-0400 I CONTROL (initandlisten) modules: none
2020-09-11T14:34:18.784-0400 I CONTROL (initandlisten) build environment:
2020-09-11T14:34:18.784-0400 I CONTROL (initandlisten) distarch: x86_64
2020-09-11T14:34:18.784-0400 I CONTROL (initandlisten) target_arch: x86_64
2020-09-11T14:34:18.785-0400 I CONTROL (initandlisten) options: { config: “C:Program FilesDELLWMSMongoDBmongod.cfg”, net: { bindIp: “localhost”, port: 27017 }, security: { authorization: “enabled” }, service: true, storage: { dbPath: “C:Program FilesDELLWMSDatabaseMongo” }, systemLog: { destination: “file”, path: “C:Program FilesDELLWMSMongoDbmongod.log” } }
2020-09-11T14:34:18.785-0400 I – (initandlisten) Detected data files in C:Program FilesDELLWMSDatabaseMongo created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2020-09-11T14:34:18.786-0400 I STORAGE (initandlisten) wiredtiger_open config: create,cache_size=3583M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2020-09-11T14:34:18.797-0400 E STORAGE (initandlisten) WiredTiger error (0) (1599849258:797507)(1816:140721228962256), file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 1919248233 doesn’t match expected checksum of 3720300442
2020-09-11T14:34:18.797-0400 E STORAGE (initandlisten) WiredTiger error (0) (1599849258:797507)(1816:140721228962256), file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2020-09-11T14:34:18.797-0400 E STORAGE (initandlisten) WiredTiger error (-31804) (1599849258:797507)(1816:140721228962256), file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-09-11T14:34:18.797-0400 I – (initandlisten) Fatal Assertion 28558 at srcmongodbstoragewiredtigerwiredtiger_util.cpp 361
2020-09-11T14:34:18.797-0400 I – (initandlisten)

***aborting after fassert() failure

Thank You,

javascript – Basic REST API for manipularing a MongoDB collection, using Node, Express, and Mongoose

I’m practicing back-end programming and NodeJS. As an exercise, I’m attempting to build a REST API for a MongoDB collection. I’m also learning to use the Express and Mongoose middle-wares, so that’s what we’ll use for the server and database respectively. Also practicing async / await to handle promises.

The requirements for this basic REST API and exercise are:

  • Support get and delete on individual resources.
  • Support get and post on the resource collection.
  • Apply generalization and separation of concerns.
  • Protect against Mongo injection.
  • Use async / await to handle promises.

This is the current working implementation:


app.js

const express = require('express')
const mongoose = require('mongoose')
const morgan = require('morgan')

const songRouter = require('./routes/song-router.js')

const mongurl = 'mongodb://localhost:27017/library'
const port = 3000

const app = express()
app.use(morgan('combined'))
app.use('/songs', songRouter)

mongoose.connect(mongurl, () => {
  console.log(`n    >> Mongoose connected to ${mongurl}`)
})

app.listen(port, () => {
  console.log(`n    >> Node listening to port ${port}`)
})

models/song-model.js

const mongoose = require('mongoose')

const song = {
  name: {
    type: String,
    required: true
  },

  author: {
    type: String,
    required: true
  },

  key: String
}

const options = {
  timestamps: true
}

const schema = new mongoose.Schema(song, options)

module.exports = mongoose.model('song', schema)


routes/song-router.js

const express = require('express')

const control = require('../controllers/song-control.js')

const router = express.Router()
router.use(express.json())

router
  .route('/')
  .get(control.getAll)
  .post(control.postOne)

router
  .route('/:songId')
  .get(control.getOne)
  .delete(control.deleteOne)

module.exports = router


controllers/song-control.js (version 1, without generalization)

const songModel = require('../models/song-model.js')

exports.getAll = async (req, res, nxt) => {
  try {
    const allSongs = await songModel.find({})
    res.status(200).json(allSongs)
  } catch (err) {
    nxt(err)
  }
}

exports.getOne = async (req, res, nxt) => {
  try {
    const oneSong = await songModel.findById(req.params.songId)
    res.status(200).json(oneSong)
  } catch (err) {
    nxt(err)
  }
}

exports.postOne = async (req, res, nxt) => {
  try {
    const postedSong = await songModel.create(req.body)
    res.status(200).json(postedSong)
  } catch (err) {
    nxt(err)
  }
}

exports.deleteOne = async (req, res, nxt) => {
  try {
    const deletedSong = await songModel.findByIdAndDelete(req.params.songId)
    res.status(200).json(deletedSong)
  } catch (err) {
    nxt(err)
  }
}


controllers/song-control.js (version 2, first attempt at generalization)

const songModel = require('../models/song-model.js')

exports.getAll = buildMongoFunction('find')

exports.getOne = buildMongoFunction('findById', true)

exports.postOne = buildMongoFunction('create', false)

exports.deleteOne = buildMongoFunction('findByIdAndDelete', true)

function buildMongoFunction (funName, argIsParam) {
  return async (req, res, nxt) => {
    const arg = argIsParam ? req.params.songId : req.body

    try {
      const reply = await songModel(funName)(arg)
      res.status(200).json(reply)
    } catch (err) {
      nxt(err)
    }
  }
}



I’m looking forward to all kinds and types of feedback: style, bugs, anti-patterns, ways to do this more concise / maintainable / redeable, conventions, best practices; whatever you think can be improved, please share.

I have some specific questions:

  • The generalization of controllers/song-control.js feels hacky. Is there a better way to implement the generalization of that pattern? How’d you do it?

  • How well are these concepts being applied: generalization, separation of concerns? Would you separate responsibilities even further? Or are they too separated? Can something be further generalized?

  • How well is async / await being used?

  • Should I sanitize inputs? Or is enforcing Mongoose models and schemas protection enough against Mongo injections?

  • Seems that Mongoose queries do not return promises. Is the async / await code here doing any actual asynchronous job?

  • What would you recommend doing in a different way?

mongodb – Mongo Schema for News and Comments + Comment for Comments

I’m working a Facebook kind of application where I’ll have a News item for which comments will be there. Its also having comment for comments option. Its like tree of comments.

I’m creating my schema like below:

var newsSchema = mongoose.Schema({
  description: { type: String, required: 'Provide the description' },
  category: { type: String, enum: Constants.categories, required: 'Kindly select the category to which the news belongs' }
});


var commentSchema = mongoose.Schema({
  text: { type: String, required: 'Kindly enter the comment text' },
  newsId: { type: mongoose.Schema.Types.ObjectId, ref: 'News', required: 'Provide the news ID to which this comment belongs' },
});

If its just News item with comments, I would have created an array inside my “newsSchema”. But how can I maintain “comment to a comment” and still maintain the reference.

Looking for some reference or idea to handle.

node.js – Como separar a logica da interface em consultas com mongodb e nodejs

estou estudando o Mongodb e o NodeJs, estou iniciando . Baixei um codigo de exemplo do git e comecei a personalizar. O codigo original da consulta era assim:

router.get('/products/:page', (req, res, next) => {
                 
   let perPage = 9;
   let page = req.params.page || 1;

   Product
     .find({}) // finding all documents
     .skip((perPage * page) - perPage) // in the first page the value of the skip is 0
     .limit(perPage) // output just 9 items
     .exec((err, products) => {
       Product.count((err, count) => { // count to calculate the number of pages
         if (err) return next(err);
         res.render('products/products', {
           products,
           current: page,
           pages: Math.ceil(count / perPage)
         });
       });
     });
});

Porem preciso separar a logica da consulta do router, entao criei este fonte:

const Product = require('../produto/product');


exports.ListaProdutos = async (req, res, next) => {
        let perPage = 9;
        let page = req.params.page || 1;
      
        await Product
          .find({}) // finding all documents
          .skip((perPage * page) - perPage) // in the first page the value of the skip is 0
          .limit(perPage) // output just 9 items
          .exec((err, products) => {
            Product.count((err, count) => { // count to calculate the number of pages
              if (err) {
                  return next(err)}
              else {
                 res.render('products/products', {
                 products,
                 current: page,
                 pages: Math.ceil(count / perPage)
               })
            };
            });
          });
};

E o meu router alterei par este codigo:

router.get(‘/products/:page’, (req, res, next) => {
ProdutoDAL.ListaProdutos(req, res, next);

});

Continua funcionando perfeitamente. Porem se notarem dentro do método Listaprodutos ainda respondo a consulta com um res.render(‘products/products’.
Eu não gostaria que ficasse assim. Eu gostaria que o método Listaprodutos retornasse apenas os dados
e o router apenas testasse se o retorno tem dados e executasse o res.render.

Como faço isto?
Eu tentei :

if (err) {
   return next(err)}
else {
   return products;
     };

Mas desta forma não sei como tratar o retorno da função no router.
E tem outro agravante que eh a paginação que o código original implementa. Como vou passar para o router as variáveis: perpage e page?

Outra pergunta: Está certo o que estou tentando implementar?

Enfim, estou mais perdido que cego em tiroteio. Preciso aprender, mas quero aprender do jeito certo.

Obrigado a todos.

nosql – Upsert unique index Mongodb?

I’ve written ETL in MongoDB which processes data from multiple collections and insert the processed data in new collection. While inserting processed data to new collection, I get Duplicate key Error E11000 due to reprocessing of some already existing records and _id conflicts with existing. How can I upsert these duplicate documents??

I’m using this:

var data = db.getCollection("col1").find().toArray()
db.ProcessedData.insert(data,{upsert : true})

But no use.
I don’t want to loop through all the records in col1 to insert one by one as it has large number of docs every run.