Saturday, July 24, 2021

New in Mongoose 5.10: Optimistic Concurrency - Safe Save Conflict

Optimistic concurrency is a better way to handle save conflict compare to the article I wrote previously.

https://scalaoncloud.blogspot.com/2021/03/how-to-avoid-save-conflict-in-mongoose.html


Optimistic concurrency tracks the version of the document and increments the version every time save(). If the version of the document in the database changed between when you loaded the document and when you save(), Mongoose will throw an error.

https://mongoosejs.com/docs/guide.html#optimisticConcurrency

https://thecodebarbarian.com/whats-new-in-mongoose-5-10-optimistic-concurrency.html 

Thursday, April 22, 2021

Create Node.js & Express.js with Typescript Project

    Here are the steps to create Node.js & Express.js with Typescript Project.

    1. Create a folder.
    2. Type command "npm init" to create package files.
    3. Type "npm install -g typescript" to install typescript.
    4. Type "tsc --init" to create typescript config file.
    5. Type "npm install express --save" to download express module.
    6. Type "npm install nodemon --save-dev" to download nodemon developer tool.
    7. Type "install install @types/node --save-dev" to download typescript node definition files.
    8. Type "install install @types/express --save-dev" to download typescript express definition files.
    9. Edit the package.json
    10. Change to main to "dist/app.js" to point to the start javascript file.
    11. Add the start scripts, "start" : "nodemon dist/app.js"
    12. The package.json will look like below:
    13. Edit the tsconfig.json
    14. Add "moduleResolution" : "node" under "module" : "commonjs".
    15. Change the "outDir" : "./dist". This is the location where typescript compiles the javascript.
    16. Change the "rootDir" : "./srv". This is where we store the typescript files.
    17. The tsconfig.json will look like below:
    18. Create /src folder and add app.ts. Edit the app.ts with the below script.
      import express from "express";

      const app = express();
      const port = process.env.port || 3001;

      // Adding middleway to use syntax req.body.name.
      app.use(express.json());

      app.get("/", (reqres=> res.send("Hello World!"));

      app.listen(port, () =>
        console.log(`Example app is listening on post ${port}!`)
      );

    19. Open a terminal run "npm start". This will start the web server using nodemon.
    20. Open another terminal run "tsc --watch". This will watch the /src folder to compile the ts to javascript.
    21. Call http://localhost:3001/ using postman. If you see "Hello World!", congratulation, you have completed the setup Node.js & Express.js with the Typescript project.

    Below is the project you may download from GitHub.

    Below GitHub is the example of the Node.js & Express.js project. You may compare the different of project setup.

    Friday, April 16, 2021

    Typescript: Shorthand Initialization

    In a lot of use cases, the fields are initialized in the constructor.

    class Person {

        // Fields declaration 
        name : string;
        private id : string;

        // Initial the field value in constructor
        constructor(name : stringid : string) { 
            this.name = name;
            this.id = id;
        }

    }

    You can simplify to

    class Person {

        constructor(private name : stringpublic id : string) {}

    }

    Typescript: Singleton

     Singleton is a design pattern that ensures that only one objects exits.

    /**
     * The Singleton class defines the `getInstance` method that lets clients access
     * the unique singleton instance.
     */
     class ExampleSingleton {
        private static instanceExampleSingleton;

        /**
         * The Singleton's constructor should always be private to prevent direct
         * construction calls with the `new` operator.
         */
        private constructor() { }

        /**
         * The static method that controls the access to the singleton instance.
         *
         * This implementation let you subclass the Singleton class while keeping
         * just one instance of each subclass around.
         */
        public static getInstance(): ExampleSingleton {
            if (!ExampleSingleton.instance) {
                ExampleSingleton.instance = new ExampleSingleton();
            }

            return ExampleSingleton.instance;
        }

        /**
         * Finally, any singleton should define some business logic, which can be
         * executed on its instance.
         */
        public someBusinessLogic() {
            // ...
        }
    }

    /**
     * The client code.
     */
    function clientCode() {
        const s1 = ExampleSingleton.getInstance();
        const s2 = ExampleSingleton.getInstance();

        if (s1 === s2) {
            console.log('Singleton works, both variables contain the same instance.');
        } else {
            console.log('Singleton failed, variables contain different instances.');
        }
    }

    clientCode();

    Singleton used case can be logging class where we can re-use the object.

    ES2019 Added Class Private Access Modifier

    ES2016 had introduced class without the access modified. Refer to my below article. 

    https://scalaoncloud.blogspot.com/2016/12/javascript-evolution-esmascript-6.html


    ES2019 has new Class Access Modified. You can add a private modified with # infont. For example:

    class SponsorWidget {
      constructor(namedescriptionurl) {
        this.name = name;
        this.description = description;
        this.url = url;
      }

      // Private method
      #buildURL(url) {
        console.log(url);
      }

      render() {
        // Calling private method
        this.#buildURL(this.url);
      }
    }

    // Here is how we involve the class
    let sponsorWidget = new SponsorWidget(
      "Google",
      "Search",
      "https://www.google.com"
    );
    sponsorWidget.render();

    // Error throw.
    sponsorWidget.#buildURL("https://www.hotmail.com");

    Wednesday, April 7, 2021

    Why does Prettier not format code in VS Code?

    I was trying to set up Prettier in my VS Code and not sure why it does not format code. After googling I found the fix.

    1. Goto "VS Code > View > Command Palette".

    2. Search for "Format Document With".

    3. Click on "Configure Default Formatter".

    4. Click on "Prettier - Code Formatter".

    I had recommended "Indent Rainbow" on my previous blog on "What is your first VS Code plugin?". Well, this plugin is not that useful if you have Prettier. Prettier already indent your code and I don't need indent colour.

    Sunday, April 4, 2021

    Flatten array in MongoDb ($unwind) vs HCL Notes (View)

     Let's say you have an array dataset:

    [{

        "name": "Simon",

        "hobbies": ["badminton", "tennis"]

    },{

        "name": "Max",

        "hobbies": ["swimming", "tennis"]

    }]


    You can fatten the hobbies array to multiple documents using $unwind.


    db.hobby.aggregate([{$unwind : "$hobbies"}]).pretty()


    The result:

    [{

            "_id" : ObjectId("6066d4146fc1865ad270a066"),

            "name" : "Simon",

            "hobbies" : "badminton"

    },

    {

            "_id" : ObjectId("6066d4146fc1865ad270a066"),

            "name" : "Simon",

            "hobbies" : "tennis"

    },

    {

            "_id" : ObjectId("6066d42b6fc1865ad270a067"),

            "name" : "Max",

            "hobbies" : "swimming"

    },

    {

            "_id" : ObjectId("6066d42b6fc1865ad270a067"),

            "name" : "Max",

            "hobbies" : "tennis"

    }]


    Let produce the same data set in HCL Notes.


    In the columns property, set "Ascending" sort and check to "Show multiple values as separate entries".


    The result:



    After splitting the array data into a different document, we can group them to get to the sum. For example, we would like to get to the total number of each hobby. 

    In MongoDb, here is the query:

    db.hobby.aggregate([
    {$unwind : "$hobbies"},
    {$group: { _id : '$hobbies',count: { $sum: 1 }}}
    ]).pretty()

    The result:

    { "_id" : "badminton", "count" : 1 }
    { "_id" : "tennis", "count" : 2 }
    { "_id" : "swimming", "count" : 1 }

    In HCL Notes, category (group) the hobbies column. 


    Add a new column with below setting:


    The result:


    Wednesday, March 31, 2021

    What you should know on MongoDb Indexes

     

    32MB Sort Limit in MongoDB

     MongoDb stores the search result in memory before sort. It has a 32MB memory limit. 


    The workaround is to create an index on the search field.

    Implementing Search feature using Text Index in MongoDb

    Let's say we have 1,000,000 documents with below data schema:

    [{

    name : "Simon",

    comment: "This is awesome article. Max also pretty good on this.",

    score: 29.52711235568646,

    age: 40

    }, 

    {

    name : "Max"

    comment: "You are an awesome person.",

    score:47.148964115304345,

    age: 72

    }]


    We want to implement a search feature where user can search the name and comment. The search weight of the name is higher than the comment. For example, when the user searches for Max on the above 2 documents, the search result will return both documents. However, the document with Max name will show first.


    MongoDb has "Text Search" index that performs a text search of string content. The index can easily create using below command:


    db.ratings.createIndex( 

    { name: "text", comment: "text" },

    { weights: { name: 10, comment: 5 }} 

    )


    When creating the index, the collection will be locked, you can't find, update, etc on the collection. You can create the index on the background without lock the collection using the below command. Creating an index background is slower.


    db.ratings.createIndex( 

    { name: "text", comment: "text" },

    { weights: { name: 10, comment: 5 }},

    { background: true }

    )


    Let search Max using the below query:


    db.ratings.find( { $text: { $search: "Max" } } )


    To display the search score and sort based on the search score, you can use below query:


    db.ratings.find( 

    { $text: { $search: "Max" } }, 

    { searchScore: { $meta: "textScore" }} 

    ).sort( 

    { searchScore: { $meta: "textScore" }} 

    ).pretty()


    Here is the result:

    [{

    name : "Max"

    comment: "You are an awesome person.",

    score:47.148964115304345,

    age: 72,

    searchScore: 11

    },{

    name : "Simon",

    comment: "This is awesome article. Max also pretty good on this.",

    score: 29.52711235568646,

    age: 40,

    searchScore: 5.5

    }]


    References:

    Tuesday, March 30, 2021

    How to update an array in MongoDb using $ Operator?

     Let's say we have below dataset:

    [{

    name : "Simon"

    hobbies : [

    { title : "badminton", frequency : 2 } ,

    { title : "Swimming", frequency : 1 } 

    ]

    }, 

    {

    name : "Max"

    hobbies : [

    { title : "badminton", frequency : 1 } ,

    { title : "Swimming", frequency : 2 } 

    ]

    },]


    $ is a placeholder to update the first element that matches the query condition in the array. For example, below change Simon's hobby from Badminton to Tennis and frequency from 2 to 5. 


    db.hobby.updateMany(

        {"hobbies" : {$elemMatch: {title : "badminton", frequency : 2}}}, 

        {$set : {"hobbies.$" : {title : "Tennis", frequency : 5 }}}

    );


    You can use $[].<fieldname> to update all element in the array. For example, add expert into all hobbies.


    db.hobby.updateMany(

        {}, 

        {$set : {"hobbies.$[].expert" : true }}

    );


    You can add a new element into the array by adding $.<fieldname>. For example, adding expert new field:


    db.hobby.updateMany(

        {"hobbies" : {$elemMatch: {title : "Tennis", frequency : 5}}}, 

        {$set : {"hobbies.$.expert" : true }}

    );


    Reference:
    https://docs.mongodb.com/manual/reference/operator/update/

    Monday, March 29, 2021

    How to find exact embedded documents using $elemMatch?

     Let's say we have below dataset:

    [{

    name : "Simon"

    hobbies : [

    { title : "badminton", frequency : 2 } ,

    { title : "Swimming", frequency : 1 } 

    ]

    }, 

    {

    name : "Max"

    hobbies : [

    { title : "badminton", frequency : 1 } ,

    { title : "Swimming", frequency : 2 } 

    ]

    },]


    Our goal is to find a person who likes badminton and 2 badminton playings frequently.


    If you using the below query, it will return Simon and Max.


    db.people.find({"hobbies .title":"badminton","hobbies.frequency ":2});

    db.people.find({$and: [{"hobbies .title":"badminton"},{"hobbies .frequency ":2}]})


    Max returned in the search result because he has a badminton hobby and frequency 2 in swimming.


    To search for the exact person who likes badminton and 2 badminton playing frequently, you can use $elemMatch operator.


    db.people.find({"hobbies":{$elemMatch: {"title":"badminton","frequency ":2}}})

    When we need to use $And in MongoDb Query?

     By default, MongoDb has the $and operator in find query. 


    For example of this:

    db.products.find({$and: [{name:"Mazda 3"}, {manufacturer:"Mazda"}]});


    You can shortcut to:

    db.products.find({name:"Mazda 3", manufacturer:"Mazda"});


    So why still $and operator?


    MongoDb builds using Javascript engine, when you search the same fields, the last search criteria will overwrite all. For example below


    db.products.find({parts:"Seats", parts:"Doors"});


    is equivalent to


    db.products.find({parts:"Doors"});


    The correct way to search the same fields is using $and operator.


    db.products.find({$and:[{parts:"Seats"}, {parts:"Doors"}]});

    Thursday, March 25, 2021

    How to avoid save conflict in mongoose?

    Save conflict is when 2 persons are saving the same document. For example, Person A and Person B open the same document. Person A saves the document, followed by Person B, then Person A's data over-write by Person B.


    To avoid this, you can use either version check or document lock mechanism.


    Version Check:

    By default, when you save the document using mongoose, mongoose saves the version number in __v field. We can use the field to do version check to avoid save conflict.


    Person A opens and edits the version 1 document. Meantime, Person B also opens the edit version 1 document.


    Person A success save the document using below query:

    posts.update({ _id: postId, __v: versionNumber } , { $set: { 'comments': updatedText }})


    When Person B try to save the document using the same query, the save will be failed because of the wrong version number.


    Document Lock:

    Locking when Person A edit the document, saved a lock flag to the document.


    When person B tries to edit the same document, the system will prompt saying the document is currently edit (lock) by Person A.


    Bonus:

    If you don’t want to use versioning in your schema you can disable it by passing false for the versionKey option. 

    schema.set('versionKey', false);

    Wednesday, March 24, 2021

    What if the ref document deleted from the backend in Mongoose's Populate Ref?

    Mongoose's populate joins 2 collections by using the ref. 

    What if the ref document deleted from the backend?

    You still can see the ref when you searching using MongoDb driver and MongoDb Compass. However, you won't see the ref when you search using mongoose.


    Thursday, March 18, 2021

    update() vs updateMany()

    To avoid accidents replace all fields in many documents, I would suggest do not use update(). Use below shell methods instead:

    replaceOne() : replace 1 document.

    updateOne() : update 1 document on specific fields.

    updateMany(): update many document on specific fields.


    Good Example:

    db.flight.updateOne({depart:"KLIA"}, {$set:{arrivaltime:"3pm"}})

    db.flight.updateMany({depart:"KLIA"}, {$set:{arrivaltime:"3pm"}})

    db.flight.replaceOne({depart:"KLIA"}, {arrivaltime:"3pm"})


    Bad Example:

    db.flight.update({depart:"KLIA"}, {arrivaltime:"3pm"})

    Thursday, February 18, 2021

    JavaScript Features That Are Introduced In ES2021

    1. String.prototype.replaceAll
      A new string method to replace all occurrences items.
      const catPhrase = 'A cat sat on the cat mat';
      const dogPhrase = catPhrase.replaceAll('cat', 'dog');
      console.log(dogPhrase); // A dog sat on the dog mat
      

    2. Logical Assignment Operators

      - Or Or Equals

      Or Or Equals provides a nice shorthand for us to provide an alternative value if the initial value is empty. 

      In the first example below, as a is truthy, a remains equal to “hello”.
      let a = “hello”;
      a ||= 10;
      console.log(a); // hello
      
      In the second example, found below, as b is falsey the value is then set to “world”.
      let b = “”;
      b ||= “world”;
      console.log(b); // world
      - And And Equals
      And And Equals provides a nice shorthand for us to override a value if a previous value was defined.

      In the first example below, as a is truthy, a is then set to “hello”.
      let a = true;
      a &&= “hello”;
      console.log(a); // hello
      In the second example below, as b is falsey, b remained equal fo false.
      let b = false;
      b &&= “world”;
      console.log(b); false

    3. Numeric Separators
      Numeric Separators introduces the separators for numeric literals with the intent to make them easier to read. Before ES2021 we would write our million value as follows:
      const billion = 1000000000
      With Numeric Separators we can make the number much easier to read by introducing underscores.
      const billion = 1_000_000_000

    4. Private Methods and Fields
      ES2021 finally fixes this omission by introducing private methods and private fields. To indicate a method or field is private you simply need to use a hash (#) before the name.
      class ClassWithPrivateField {
       #privateField
      }
      
      class ClassWithPrivateMethod {
       #privateMethod() {
       return 'hello world'
       }
      }
      
      class ClassWithPrivateStaticField {
       static #PRIVATE_STATIC_FIELD
      }
      Class was introduced on ES6, you may refer to below article:
      https://scalaoncloud.blogspot.com/2016/12/javascript-evolution-esmascript-6.html

    Wednesday, February 17, 2021

    Nodejs Data Caching with Redis Part 3: Performance Testing

    This testing is running on 12GB RAM Window PC. NodeJS, MongoDB and Redis are installed on standalone PC.

    The test is querying 10,000 documents.

    It takes 1005ms response time query from MongoDB Server.


    40% faster where 594ms response time from Redis Cache.



    Here is all about Redis Cache.

    Are you interested in live scoreboard build with Redis Sorted Set and Socket Programming? Feel free to let me know, probably my next sharing.

    Tuesday, February 16, 2021

    Nodejs Data Caching with Redis Part 2: Ultimate Way

    I have shown an example of a direct approach on how to use Redis on NodeJS in Part 1. The disadvantage of the direct approach is redundant code where you have to initial Redis Server and to check whether there is a cache whenever you use it.

    In this part 2, I am going to centralise the intitial Redis Server and add new method in mongoose:

    .cache() = Cache value into redis memory database.

    clearCache() = Delete value from redis memory database.

    You can download the code at https://github.com/htsiah/node-redis-mongodb

    The Redis Code store in utils/useCacheUtil.js and the sample code at controllers/TeacherController.js

    Wednesday, February 10, 2021

    Nodejs Data Caching with Redis Part 1: Caching in Action

    You have to install Redis, Another Redis Desktop Manager and NPM Redis for this example.

    Install Redis on Linux

    Install Redis on Window

    Install free version Another Redis Desktop Manager to view the cache data. 

    Install NPM Redis
    npm install redis

    This is how you connect redis in NodeJS:
    const redis = require('redis')
    const redisURL = 'redis://127.0.0.1:6379'
    const client = redis.createClient(redisURL)

    For basic value, you can use set and get method. The first parameter is the search key and second parameter value stored.

    // Store 'hi' as search key, and value is 'there'.
    client.set('hi','there');

    // Output: there
    client.get('hi', ( err, val) => { console.log(val) } )

    Redis support hset where you can have 2 search keys. Generally, I think it is practical to use hset for mongodb where the search keys are collection name and search query.



    You may refer example on CRUD in StudentController:

    Create - http://localhost/api/student
    - When there is new document, delete collection in the cache.
    redisClient.del(StudentModel.collection.collectionName)


    Read - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3
    - If there is cache data, return cache data.
          const cacheStudent = await redisClient.hget(
            StudentModel.collection.collectionName
            JSON.stringify(query.getQuery())
          )

          if (cacheStudent) {
            // Remember convert to javascript object
            console.log('Get from cache.')
            return res.json(JSON.parse(cacheStudent)); 
          }  
    - If not, find the data in mongodb, store the data to redis and return the data.
          const student = await query.exec();

          redisClient.hset(
            StudentModel.collection.collectionName
            JSON.stringify(query.getQuery()), 
            JSON.stringify(student.toObject({ getters: false }))
          )

    Update - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3
    - Same as create, delete the collection.

    Delete - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3
    - Same as create, delete the collection.

    The above is an example using collection as key. The different requirement may use different key.

    Sunday, February 7, 2021

    Transactions With Mongoose & NodeJs, Express

    Local Machine Implementation:

    Transactions only work on replicate sets. So, the first step is to convert your standalone MongoDb instance to single replicate set. 

    1. Shutdown MongoDB Server.
    2. Add below entry to mongod.cfg
      replication:
        replSetName: "rs0"
    3. Start MongoDb Server.
    4. Initiate the replica set from within the mongo shell using command "rs.initiate()".

    After converting to replicate set, we can work on the NodeJs code.


    Docker Implementation:

    1.Create a dockerfile: mongo.dockerfile
    FROM mongo
    RUN echo "rs.initiate();" > /docker-entrypoint-initdb.d/replica-init.js
    CMD ["--replSet", "rs0"]

    2. Installation:
    2.1 Build the image
    docker build -t custom-replica-mongo . -f mongo.dockerfile`
    2.2 start the instance
    docker run custom-replica-mongo

    3. Create with docker-compose
    version: '3'
    services:
      mongo:
        image: custom-replica-mongo
        build:
          context: .
          dockerfile: mongo.dockerfile
        ports:
         - 27017:27017


    Create a model: 

    const StudentModel= mongoose.model('StudentModel', new mongoose.Schema({

      year: { type: Number },

      standard: { type: Number },

    }));

    Example of saving a document using transaction:

    const createStudent = async (req, res, next) => {

        const { year, standard } = req.body;


        const session = await StudentModel.startSession();

        session.startTransaction();


        const newStudent = new StudentModel({

          id: uuidv4(),

          year: year,

          standard: standard,

        })

      

        try {

          const opts = { session };

          await newStudent.save(opts);


          // commit the changes if everything was successful

          await session.commitTransaction();

          session.endSession();

        } catch (err) {

          // If an error occurred, abort the whole transaction and

          // undo any changes that might have happened

          await session.abortTransaction();

          session.endSession();


          const error = new Error(' Creating document failed.');

          return next(error);

        }

      

        res.status(201).json(newStudent);

    };

    For full example of transaction in MVC, you can refer below repository.

    The example included updating, delete in transaction.

    References:
    https://docs.mongodb.com/manual/tutorial/convert-standalone-to-replica-set/

    Friday, January 8, 2021

    Improve NodeJS Performance using Cluster and PM2

    NodeJS is single thread, meaning a request must be completed before it can execute another request. This example demonstrates how inefficient in NodeJS: https://github.com/htsiah/node-performance-using-cluster/tree/main/blocking-the-event-loop

    To improve the performance, we can implement cluster where you can fork as many as child processes. The rule of thumb is the child can not more than logical CPU. You can download the cluster implementation here. https://github.com/htsiah/node-performance-using-cluster/tree/main/cluster-in-action

    This article explains the number of child process and thread vs the logical CPU: https://dev.to/johnjardincodes/increase-node-js-performance-with-libuv-thread-pool-5h10

    PM2 is a NodeJS cluster package. This package used in many production environments. I recommend  PM2 instead of reinventing the wheel. The above example is to let you understand how the cluster works.

    First, you need to install PM in the global environment by using the command: npm install -g pm2

    To start the application using pm2: pm2 start app.js -i -1 (-1 is maximum child process.)

    To show all the process: pm2 list

    To delete the process: pm2 delete app

    More refer to https://www.npmjs.com/package/pm2