Initial motivation

[denny-il]

The motivation behind Neemata:

I’ve been working with different technologies and frameworks, like Express, Fastify, Nest, tRPC, Django, Flask, gRPC, Laravel, and some other. Most of them provide very nice experience for solving some specific tasks. Some of them, like Nest, try to solve virtually every possible problem you could come across. Some of them require you to follow “their” way to do things. Some of them are pretty much simple web servers, providing little abstraction on top of HTTP.

For some context

Most projects, I’ve been working on, were not well defined in terms of business requirements. Some of them were startups. Really often I receive some new requirements just the few days after I receive an approval to start work on some feature/changes (sometimes after long discussions). Unfortunately, that’s my reality and what I have to work with.
Also, I’ve found out that most projects I’ve been working on have quite a lot of similar necessities:

1. Some endpoints to serve/mutate data with input/output validation/serialisation.
2. Some binary data handling, e.g file uploads/downloads.
3. Some job cron/scheduler.
4. Some delayed task runner.
5. Some command line access to application context.
6. Some server-to-client events or even full-fledged bidirectional communication.
7. Some way to go beyond web only.

Why not just use Nest?

Complete, backed by huge community, battle-proofed architectural approaches.

1. Enormous amount of boilerplate (if you do it “Nest” way). After few month of development you end up with hundreds of files, classes and literally thousands of meaningless lines code like imports/exports/deps and so on. In my experience navigating through that hits dev performance by a lot, and it the biggest downside for me.
2. No built-in multi-threading and vertical scaling. Yes, you can use node:cluster, but simply spin a new process is a bit different from proper multi-threading.

In the end, in my opinion, Nest is very nice choice for larger teams, with larger amount of resources, and well-defined business requirements and good engineers on board to build complex solutions. It has some way/solution to wide spectrum of problems you could come across, as well as quite impressive amount of first-party integrations with other technologies/frameworks/services. But it still is a disaster for small teams and product MVPs, as the development speed slows down dramatically.

Why not Fastify then?

Lightweight, backed by great open source developers like Matteo Collina, and large community.

1. Lack of extendability in terms of transports support. Most things are done around http and req/res.
2. Same as №2 for Nest.

Then just use tRPC, goddamnit.

1. No streaming support.
2. Same as №2 for Nest.
3. Extremely subjective, but source code looks too... messy?. Alright, don't count this one

Why use JS at the first place?

Well, because it is a very flexable, multi paradigm language. Due to its loosey goosey nature, it suits very good for prototyping and MVPs.

So, here the things I’d like to have from my tool:

1. Start quick, and stay quick: an ability to move/change/update/refactor quickly in environment with not well-defined business requirements.
2. Extendability/Scaling: environment/infrastructure/transport agnostic approach to write code, so future changes of those aspects should not require any changes to application code.
3. DX: I hate and love JS at the same time. Different module systems (ESM, CJS), tsconfig madness, using one javascript to build another javascript into some third javascript (hi, babel). All of these things definitely add nothing good to developer experience, and that has to be taken care of, so the app just runs.

How Neemata solves (or plans to solve) these problem:

1. Environment and transport agnostic design. Everything is (will be) designed with such mindset.
2. Built-in multi-threading support, so you don't have to fight with the framework, and/or you don't have to touch that nasty k8s just yet. Keep using cores while you can - they are much cheaper!
3. Minimizing boilerplate by using declarative loaders and code generation. Giving up on this: Introduce modules #104
4. More in Roadmap #49

Who could be interesting in Neemata? (Theoretical story of your success)

Project where you have to move fast, because it still "finding itself". At the beginning you just need to quickly bootstrap simple API endpoints with basic input validation. You add a web client also written with JS/TS, in this case you can just statically type the whole thing. Meanwhile it've grown to the degree when you need to execute some heavy (for an APIWorker) tasks, and you just spin new TaskWorker by changing one config option. At some point of time you also need to add bidi communication, and you just add additional WebsocketsTransport, and now it just works with websockets as well. You just provider a SubscriptionManager, define your server-side events and you good to go. After some time your project have grown a bit, and it seems like only one APIWorker is not enough anymore, and you just spin another one, again, just by changing config option. But you do not worry that some clients can connect to one worker, and some to another, therefore requiring some external message broker for events routing/propagation, because the app still runs within one machine and everything you do is just swap SubcriptionManager with the appropriate one provider by Neemata. Сatching a wave of success you now also need mobile app, fortunately, Neemata got you covered, because you can generate API interfaces for the native platform using JSONSchema, that can be generated from the input/output schemas you were providing with your Procedures. You, your business and your client are very happy, because everything above was done without application code even knowing it. It just worked.

At some point you might go all-in with k8s. But with Neemata, you have delayed that time as long as you could. Probably, by that time your product found much more clear/defined usecases/requirements and it also possible, that you realized that those problems could be solved much more efficiently with another technologies or even languages, like Golang, Rust, etc.