`UploadFile()` should accept `Readable` instead of `Buffer`

[samchon]

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe it

When using @UploadFile() decorator following the NestJS guide documents, it becomes Buffer type.

As you know, the Buffer type means that backend server gathers entire data of the uploaded file. If client uploads 1GB file through the API, the NestJS backend server requires at least 1GB memory about the request. If there're 100 clients uploading the 1GB file at the same time, NestJS backend server needs 100GB memory at that time.

Describe the solution you'd like

I know that the default option of multer is using the Buffer type instance, but it seems not proper to the full framework like NestJS. If configure storage engine of the multer, the uploadaed file would be Readable stream type, and I think NestJS should do it.

Teachability, documentation, adoption, migration strategy

I have followed the https://docs.nestjs.com/techniques/file-upload document.

And very surprised by too much memory consumption.

What is the motivation / use case for changing the behavior?

I think NestJS should do like below:

1. Make streaming to be default
2. When some users want Buffer feature, they must configure by themselves
3. If hard to migrate, then have to warn those facts in the guide documents

[jmcdo29]

The thing is, it can accept Readable as you mentioned, with the right configuration. While it may be a better solution, in terms of memory management, it's also a more advanced solution in terms of Node API knowledge as not everyone is well versed with streams, and it's a breaking change at the moment. We'd have to make sure to update the docs to show how the stream could be worked with, and probably be ready to answer many, many more questions on how to use the newer approach, which would inevitably be sent to Discord or StackOverflow and answered there.

While I do think that better memory management is a good thing, there is a balance to be kept between efficient and ergonomic API for developers.

[jmcdo29]

Perhaps we can make a mention in the docs and the default of buffer and how to use streams if that is the desired approach and why you might want to do that

[samchon]

If hope to avoid breaking change, I think NestJS guide document must warn the memory problem, and give an example code converting to the streaming strategy.

[kamilmysliwiec]

Adding a mention in the docs sounds valuable

[samchon]

I'd tried to make concise @UploadFile() similar decorator supports both express and fastify, and understood why @kamilmysliwiec and @jmcdo29 had configured of @UploadFile() to just utilize the full Buffer data as default. To perform streaming, have to configure storage engine of multer, but it was not easy to determine which option to adjust as default.

On the other hand, fastify's multipart/form-data was much easier to handle. This is because streaming could be used right away without any additional settings. Therefore, I had developed @TypedFormData.Body() to utillize the full data Buffer to compatible with both express and fastify, and planning to apply memory storage to the multer side by referring to fastify's options.

Anyway, after I completed this @TypedFormData.Body() and studied it thoroughly through various experiments, I will contribute to Nest Document. The next PR may be the next month.

• https://nestia.io/docs/core/TypedFormData/
• https://github.com/samchon/nestia/blob/master/packages/core/src/decorators/TypedFormData.ts
• https://github.com/samchon/nestia/blob/master/packages/core/src/programmers/TypedFormDataBodyProgrammer.ts

[Shandur]

@samchon Have you had any chance to contribute to NestJS documentation? :)

[samchon]

Have forgotten this issue for a long time due to busy.

@Shandur Thanks for reminding. It's okay you to contribute eariler.

[saram-aman]

@samchon is the issue still in the queue or it has been closed?, if workable, assign it to me!

Thanks!

[samchon]

I'm not maintainer of NestJS, so that as you wish @saram-aman

In my case, I've solved this problem just by defining multer instance by closure function parameter like below.

https://nestia.io/docs/core/TypedFormData/

import { TypedFormData, TypedRoute } from "@nestia/core";
import { Controller } from "@nestjs/common";
import Multer from "multer";
 
@Controller("bbs/articles")
export class BbsArticlesController {
  @TypedRoute.Post()
  public async create(
    @TypedFormData.Body(() => Multer({
      storage: { ...THE STORAGE ENGINE SETTING }
    })) input: IBbsArticleCreate,
  ): Promise<void> {
    input;
  }
}
 
export interface IBbsArticleCreate {
  title: string;
  body: string | null;
  thumbnail?: File | undefined;
  files: File[];
  tags: string[];
}

[simonas88]

@saram-aman are you working on this? If not, may I take this task upon myself?

[arsalansofyrus12]

One can also mitigate customizing the Multer middleware in NestJS. Here’s an example:

import { MulterModule } from '@nestjs/platform-express';
import { diskStorage } from 'multer';
import { extname } from 'path';

@module({
imports: [
MulterModule.register({
storage: diskStorage({
destination: './uploads',
filename: (req, file, callback) => {
const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1e9);
callback(null, ${file.fieldname}-${uniqueSuffix}${extname(file.originalname)});
},
}),
}),
],
})
export class AppModule {}