File Upload Microservice - Backend

Architecture: Microservice

This project is a robust File Upload Microservice backend built with Node.js, Express, Redis, Bull, and Firebase. It includes comprehensive features for managing file uploads, queuing, and rate limiting, and is designed to handle large-scale file processing efficiently. The system integrates Firebase for cloud storage, Bull for job queuing, and Redis for rate limiting, demonstrating its scalability and production readiness. Additionally, it incorporates advanced error handling and logging to ensure reliability and maintainability.

Table of Contents

• Features
• Installation
• Configuration
• Usage
• Project Structure
• API Endpoints
• Contributing
• License

Features

• File Upload Management: Handles file uploads with support for various file types and integrates with Firebase for cloud storage.
• Job Queuing: Utilizes Bull for efficient job queuing and processing of file uploads.
• Rate Limiting: Implements rate limiting using Express Rate Limit and Redis to protect API endpoints.
• Robust Error Handling: Advanced error handling mechanisms with Winston for comprehensive logging and monitoring.
• Directory Management: Automatic directory creation and management for handling file storage.
• Scalability: Designed for scalability with integration options for message brokers like KafkaJS for future enhancements.
• JOI for error handling

Installation

1. Clone the repository:

   git clone https://github.com/reginaldsc02/file-upload-microservice.git
   cd file-upload-microservice

2. Install dependencies:

   npm install

3. Set up environment variables:

   Create a .env file in the root directory and configure the following variables:

    # Example: EXPRESS_PORT=3001
    EXPRESS_PORT=PORT_NUMBER
   
    # Example: WINSTON_SERVICE="nms-backend-microservice"
    WINSTON_SERVICE=SERVICE_NAME
   
    # Example: NODE_ENV=development
    # Example: NODE_ENV=production
    NODE_ENV=ENVIRONMENT
   
    # Example: REDIS_CLIENT_PASSWORD=enteryourredispasswordhere
    REDIS_CLIENT_PASSWORD=PASSWORD
   
    # Example: REDIS_CLIENT_HOST=enteryourredishosthere
    REDIS_CLIENT_HOST=HOST
   
    # Example: REDIS_CLIENT_PORT=30982
    REDIS_CLIENT_PORT=PORT_NUMBER
   
    # Example: CORS_ORIGIN=http://localhost:3001
    CORS_ORIGIN=ALLOWED_ORIGIN
   
    # Example: FIREBASE_API_KEY=yoursecretapikey
    FIREBASE_API_KEY=API_KEY

4. Setup Redis

• Create a Redis Account & connect to the redis cloud.

5. Start the server:

   npm run start

Configuration

The project uses environment variables for configuration. Refer to the .env.example file for the required variables. Make sure to set these variables in your .env file.

Usage

Starting the Server

To start the server in development mode:

npm run serve

If all goes well, then you should have something similar in your terminal. Refer Below:

npm run serve

> file-upload-microservice@1.0.0 preserve
> npx eslint .


> file-upload-microservice@1.0.0 serve
> nodemon ./server.mjs

[nodemon] 3.1.4
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node ./server.mjs`
info: Successfully initialized the Firebase App. {"additional":"projectId: file-upload-microservice","service":"fms","timestamp":"2024-08-17 02:31:47"}
heartbeats undefined
info: Connection to Redis Data Store have been established! (Bull.js) {"service":"fms","timestamp":"2024-08-17 02:31:47"}
info: Express Server is successfully listening! {"additional":"port: 3001","service":"fms","timestamp":"2024-08-17 02:31:47"}
info: Connection to Redis Data Store have been established! {"additional":"port: 19314","service":"fms","timestamp":"2024-08-17 02:31:48"}

API Documentation

Use a tool like Postman, Thunder Client extension in Visual Studio Code or Insomnia to test the API endpoints. Refer to the API Endpoints section for a list of available routes.

Project Structure

Please click on PROJECT_STRUCTURE to view the project structure.

Alternatively you can find the PROJECT_STRUCTURE file in the root directory of this repository and or project folder.

API Endpoints

File Upload

• POST /upload - Allows for file upload to the device

Contributing

Contributions are welcome! Please follow these steps to contribute:

1. Fork the repository
2. Create a new branch (git checkout -b feature-branch)
3. Make your changes
4. Commit your changes (git commit -m 'Add some feature')
5. Push to the branch (git push origin feature-branch)
6. Open a pull request

License

This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007

See the License for details.

Copyright

© 2024 Reginald Chand. All rights reserved.

Repository: file-upload-microservice

---

Designed & Developed with 😍 💝🌺

Kind Regards

Reginald Chand