Setting Up a Job Queue in Node.js with BullMQ and Redis

 https://docs.bullmq.io/

https://medium.com/@mjdrehman/setting-up-a-job-queue-in-node-js-with-bullmq-and-redis-in-5-minutes-0f170928c0b5

Node.js Design patterns

 https://medium.com/@debiprasaddash_35810/node-js-design-patterns-8969d9184e37

Nodejs Developer Roadmap

 https://github.com/aliyr/Nodejs-Developer-Roadmap

PM2 run different node version

To run several versions at the same time. In pm2, you can use the --interpreter options and specify the path to the node version you want.

You may install different node versions using nvm. 
Install NodeJS using Version Manager

Running direct script using PM2 process file:

module.exports = {
    apps : [{
      name   : "app1",
      cwd    : "/home/user/app1", // folder store the application
      script : "./app.js", // starting js file
      interpreter : 'node@6.9.1'    // Node version
    },
   {
      name   : "app2",
      cwd    : "/home/user/app2",
      script : "./app.js",
      interpreter : 'node@12.9.1'
   }]
}

Running using npm script using PM2 process file:

module.exports = {
    apps : [{
      name   : "app1",
      cwd    : "/home/user/app1", // folder store the application
      script : "/home/user/.nvm/versions/node/v6.9.1/bin/npm", // Folder where the npm stored
      args       : ["run", "start"],  // Argument pass to npm
      interpreter : 'node@6.9.1'    // Node version
    },
    {
       name   : "app2",
       cwd    : "/home/user/app2",
       script : "/home/user/.nvm/versions/node/v12.9.1/bin/npm",
       args       : ["run", "start"],
       interpreter : 'node@12.9.1'
    }]
}

How to Upload Files in Node.js using Multer

 https://javascript.plainenglish.io/uploading-files-in-node-js-using-multer-754526aa6817

How to secure a Node.js app?

 https://kumar-salokya007.medium.com/how-to-secure-a-node-js-app-d2581a53dddc

Using Nginx as Load Balancer with Node.js (Express)

 https://philosophyotaku.medium.com/using-nginx-as-load-balancer-with-node-js-express-63b39948f737

TS-Node: Typescript execute engine

In my previous article, I have described how to create a simple typescript nodejs project. The method works fine in a small project, but take longer compilation time on a large project because its need to compile all typescript files on every change.
Create Node.js & Express.js with Typescript Project

The solution is using ts-node where it direct run the typescript without compiling it to javascript.

1. Follow the steps to create a Node.js & Express.js with Typescript.
2. Type "npm install -g ts-node".
3. Add an entry to package.json.
   
     "scripts": {

       "test": "echo \"Error: no test specified\" && exit 1",

       "build": "tsc -b",

       "start": "node dist/app.js",

       "dev": "nodemon dist/app.js",

       "dev:ts": "nodemon --exec ts-node src/app.ts"

     },
4. Now you can just run the development using 1 console: npm run dev:ts

Best practices - API Architecture, Node.js, CI/CD

NodeJs Best Practices
https://medium.com/@quraanmaram/nodejs-best-practices-5793375c3ad8

API Architecture — Design Best Practices for REST APIs
 https://abdulrwahab.medium.com/api-architecture-best-practices-for-designing-rest-apis-bf907025f5f

10 best practices every Node.js developer must follow
https://medium.com/dhiwise/10-best-practices-every-node-js-developer-must-follow-32072950a5ac
- Further research on google styles guide and logger tools.

Basics of CI/CD
https://levelup.gitconnected.com/basics-of-ci-cd-a98340c60b04

Create Node.js & Express.js with Typescript Project

Here are the steps to create Node.js & Express.js with Typescript Project.

1. Create a folder.
2. Type command "npm init" to create package files.
3. Type "npm install -g typescript" to install typescript.
4. Type "tsc --init" to create typescript config file.
5. Type "npm install express --save" to download express module.
6. Type "npm install nodemon --save-dev" to download nodemon developer tool.
7. Type "install install @types/node --save-dev" to download typescript node definition files.
8. Type "install install @types/express --save-dev" to download typescript express definition files.
9. Edit the package.json
10. Change to main to "dist/app.js" to point to the start javascript file.
11. Add the start scripts, "start" : "nodemon dist/app.js"
12. The package.json will look like below:
13. Edit the tsconfig.json
14. Add "moduleResolution" : "node" under "module" : "commonjs".
15. Change the "outDir" : "./dist". This is the location where typescript compiles the javascript.
16. Change the "rootDir" : "./srv". This is where we store the typescript files.
17. The tsconfig.json will look like below:
    
18. Create /src folder and add app.ts. Edit the app.ts with the below script.
    
    import express from "express";

    
    

    const app = express();

    const port = process.env.port || 3001;

    
    

    // Adding middleway to use syntax req.body.name.

    app.use(express.json());

    
    

    app.get("/", (req, res) => res.send("Hello World!"));

    
    

    app.listen(port, () =>

      console.log(`Example app is listening on post ${port}!`)

    );

    
    
19. Open a terminal run "npm start". This will start the web server using nodemon.
20. Open another terminal run "tsc --watch". This will watch the /src folder to compile the ts to javascript.
21. Call http://localhost:3001/ using postman. If you see "Hello World!", congratulation, you have completed the setup Node.js & Express.js with the Typescript project.
    
    

Below is the project you may download from GitHub.

https://github.com/htsiah/simple-rest-api-using-expressjs-with-typescript

Below GitHub is the example of the Node.js & Express.js project. You may compare the different of project setup.

https://github.com/htsiah/simple-rest-api-using-expressjs

Nodejs Data Caching with Redis Part 3: Performance Testing

This testing is running on 12GB RAM Window PC. NodeJS, MongoDB and Redis are installed on standalone PC.

The test is querying 10,000 documents.

It takes 1005ms response time query from MongoDB Server.

40% faster where 594ms response time from Redis Cache.

Here is all about Redis Cache.

Are you interested in live scoreboard build with Redis Sorted Set and Socket Programming? Feel free to let me know, probably my next sharing.

Nodejs Data Caching with Redis Part 2: Ultimate Way

I have shown an example of a direct approach on how to use Redis on NodeJS in Part 1. The disadvantage of the direct approach is redundant code where you have to initial Redis Server and to check whether there is a cache whenever you use it.

In this part 2, I am going to centralise the intitial Redis Server and add new method in mongoose:

.cache() = Cache value into redis memory database.

clearCache() = Delete value from redis memory database.

You can download the code at https://github.com/htsiah/node-redis-mongodb

The Redis Code store in utils/useCacheUtil.js and the sample code at controllers/TeacherController.js

Nodejs Data Caching with Redis Part 1: Caching in Action

You have to install Redis, Another Redis Desktop Manager and NPM Redis for this example.

Install Redis on Linux

https://hackernoon.com/using-redis-with-node-js-8d87a48c5dd7

Install Redis on Window

https://divyanshushekhar.com/how-to-install-redis-on-windows-10/

https://redislabs.com/blog/redis-on-windows-8-1-and-previous-versions/

Install free version Another Redis Desktop Manager to view the cache data. 

https://github.com/qishibo/AnotherRedisDesktopManager

Install NPM Redis

npm install redis

This is how you connect redis in NodeJS:

const redis = require('redis')

const redisURL = 'redis://127.0.0.1:6379'

const client = redis.createClient(redisURL)

For basic value, you can use set and get method. The first parameter is the search key and second parameter value stored.

// Store 'hi' as search key, and value is 'there'.

client.set('hi','there');

// Output: there

client.get('hi', ( err, val) => { console.log(val) } )

Redis support hset where you can have 2 search keys. Generally, I think it is practical to use hset for mongodb where the search keys are collection name and search query.

You may refer example on CRUD in StudentController:

https://github.com/htsiah/node-redis-mongodb

Create - http://localhost/api/student

- When there is new document, delete collection in the cache.

redisClient.del(StudentModel.collection.collectionName)

Read - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3

- If there is cache data, return cache data.

      const cacheStudent = await redisClient.hget(

        StudentModel.collection.collectionName, 

        JSON.stringify(query.getQuery())

      )

      if (cacheStudent) {

        // Remember convert to javascript object

        console.log('Get from cache.')

        return res.json(JSON.parse(cacheStudent)); 

      }  

- If not, find the data in mongodb, store the data to redis and return the data.

      const student = await query.exec();

      redisClient.hset(

        StudentModel.collection.collectionName, 

        JSON.stringify(query.getQuery()), 

        JSON.stringify(student.toObject({ getters: false }))

      )

Update - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3

- Same as create, delete the collection.

Delete - http://localhost/api/student/6028c376ddf84b2bd0a6b6b3

- Same as create, delete the collection.

The above is an example using collection as key. The different requirement may use different key.

Transactions With Mongoose & NodeJs, Express

Local Machine Implementation:

Transactions only work on replicate sets. So, the first step is to convert your standalone MongoDb instance to single replicate set. 

1. Shutdown MongoDB Server.
2. Add below entry to mongod.cfg
   replication:
     replSetName: "rs0"
3. Start MongoDb Server.
4. Initiate the replica set from within the mongo shell using command "rs.initiate()".

After converting to replicate set, we can work on the NodeJs code.

Docker Implementation:

1.Create a dockerfile: mongo.dockerfile

FROM mongo
RUN echo "rs.initiate();" > /docker-entrypoint-initdb.d/replica-init.js
CMD ["--replSet", "rs0"]

2. Installation:
2.1 Build the image
docker build -t custom-replica-mongo . -f mongo.dockerfile`
2.2 start the instance
docker run custom-replica-mongo

3. Create with docker-compose
version: '3'
services:
  mongo:
    image: custom-replica-mongo
    build:
      context: .
      dockerfile: mongo.dockerfile
    ports:
     - 27017:27017

Create a model: 

const StudentModel= mongoose.model('StudentModel', new mongoose.Schema({
  year: { type: Number },
  standard: { type: Number },
}));

Example of saving a document using transaction:

const createStudent = async (req, res, next) => {
    const { year, standard } = req.body;

    const session = await StudentModel.startSession();
    session.startTransaction();

    const newStudent = new StudentModel({
      id: uuidv4(),
      year: year,
      standard: standard,
    })
  
    try {
      const opts = { session };
      await newStudent.save(opts);

      // commit the changes if everything was successful
      await session.commitTransaction();
      session.endSession();
    } catch (err) {
      // If an error occurred, abort the whole transaction and
      // undo any changes that might have happened
      await session.abortTransaction();
      session.endSession();

      const error = new Error(' Creating document failed.');
      return next(error);
    }
  
    res.status(201).json(newStudent);
};

For full example of transaction in MVC, you can refer below repository.

https://github.com/htsiah/node-redis-mongodb

The example included updating, delete in transaction.

References:

https://docs.mongodb.com/manual/tutorial/convert-standalone-to-replica-set/

Improve NodeJS Performance using Cluster and PM2

NodeJS is single thread, meaning a request must be completed before it can execute another request. This example demonstrates how inefficient in NodeJS: https://github.com/htsiah/node-performance-using-cluster/tree/main/blocking-the-event-loop

To improve the performance, we can implement cluster where you can fork as many as child processes. The rule of thumb is the child can not more than logical CPU. You can download the cluster implementation here. https://github.com/htsiah/node-performance-using-cluster/tree/main/cluster-in-action

This article explains the number of child process and thread vs the logical CPU: https://dev.to/johnjardincodes/increase-node-js-performance-with-libuv-thread-pool-5h10

PM2 is a NodeJS cluster package. This package used in many production environments. I recommend  PM2 instead of reinventing the wheel. The above example is to let you understand how the cluster works.

First, you need to install PM in the global environment by using the command: npm install -g pm2

To start the application using pm2: pm2 start app.js -i -1 (-1 is maximum child process.)

To show all the process: pm2 list

To delete the process: pm2 delete app

More refer to https://www.npmjs.com/package/pm2

Install NodeJS using Version Manager

Often Node.js can be installed with a particular Operating System's official or unofficial package manager. For instance apt-get on Debian/Ubuntu, Brew on macOs, Chocolatey on Windows. It is strongly recommended against using this approach to install Node. Package managers tend to lag behind the faster Node.js release cycle. Additionally, the placement of binary and config files and folders isn't standardized across OS package managers and can cause compatibility issues.

The popular version manager are nvm (macOS and Linux), nvm-windows (windows) and nvs (macOS, Linux and windows).

    https://github.com/nvm-sh/nvm

    https://github.com/coreybutler/nvm-windows

    https://github.com/jasongin/nvs

I prefer nvs because of cross-platform. I am a window user and my server is Ubuntu.

Comparing nvm and nvs, I have feedback from a developer saying that nvm is slower.

NodeJS - HelloWorld with ExpressJS

In this tutorial, you are going to learn how to create an express application using Express Generator and create a hello world page by creating a new route file in nodejs.

This tutorial tested on Nodejs v10.16.2 and Expressjs v4.16.1.

1. Install Express Generator. Create the application skeleton using application generator tool. Go to the project folder and type “npm install -g express-generator”

2. Create Express Project using EJS view engine. By default, Express is using PUG view engine. For this example, we are using EJS. Use this command “express --view="ejs" [folder name]” to create the express project. Example: “C:\Users\simon.siah\workspace>express --view="ejs" nodejs-expressjs-starter”.

3. Install dependencies. Type below command to install dependencies. It will read the package.json and install the stuff.

4. Start the server Type below command to start the server.

5. Open a browser and type http://localhost:3000/.

6. Route files. - The route files store in folder routes. - This section in app.js define the http routes. - Example: /, is using routes/index.js and /users us using routes/user.js.

7. View files - The view files store in folder views.

8. Create a Hello World Page.

• Copy the routes/index.js and paste into folder routes, rename the files to helloworld.js. 
• Change the content below:
  
• Create helloworld.ejs in views folder: 
• Update app.js to include helloworld route:
   
• Start the server and enter URL -> http://localhost:3000/helloworld 

Changing the EJS templates does not require a server restart, but whenever you change a JS file, such as app.js or the route files, you'll need to restart the see the changes.

I am using sublimetext as IDE to update the sources. It is free, and you can download here: http://www.sublimetext.com/ 

Here are some useful references: 

Node.js, Express.js, MongoDb Tutorial:  

https://closebrace.com/tutorials/2017-03-02/the-dead-simple-step-by-step-guide-for-front-end-developers-to-getting-up-and-running-with-nodejs-express-and-mongodb 

Node.js: 

https://nodejs.org/en/ 

https://www.w3schools.com/nodejs/default.asp 

Express.js: 

http://expressjs.com