The problem of technology choice for your new project

When an engineer starts something new (project, feature, refactoring, etc.), there is almost always a choice to pick some technology or a library to use. And nowadays, tech is a really vibrant domain, where lots of people are offering work of their minds for free in the form of open-source software. Other people commercialize their work and offer it as SaaS. Both are fine, for sure. But there is nothing harder to pick something out of many cool things, right? Moreover, there is always a hype train, pulling specific technology through the industry. And man, the temptation to jump on this train could be big!

In this post, I want to share some thoughts on how to make a good choice and what to watch out for in order to avoid bad technical decisions?

1. What happens if I choose a wrong technology?
2. What is the good choice definition then?
3. Why fancy/popular technology is not necessarily a good choice?
4. So, is conservative tech always good?
5. Where is the balance between two evils?

What happens if I choose a wrong technology?

At the very beginning, nothing bad will happen, in the worst case. You might feel a bit of friction, which could be perceived as a "learning curve," and eventually come up with some working prototype (not a product). And this is where the main danger is: in practice, there is a huge difference between a learning curve and a bad technology choice. A bad technology choice will typically have the following properties:

1. Constant TODOs and references to Stack Overflow here and there because nothing fits into the tech stack you are using
2. Constant debates about how things should be done. Sure, there might be some rocket science projects where you will have to get really creative, but for the vast majority of cases, simply following the best practices is the way to start. Every framework/platform has such a section in the documentation. Just read it carefully, and see if it makes sense to you in your specific context
3. Really hard time shaping up a mental model around the tech stack - you will not feel that you learn something new, but just that you are failing to understand the ecosystem around your product
4. Hours and hours spent in meaningless debugging of weird bugs. Always ask yourself, what exactly am I debugging? My business logic or the framework?
5. Obviously, performance or(and) costs penalties. And this is really the nasty one, as it might reveal itself later in the product lifecycle

With that said, a bad technology choice will slow down your development and likely penalize your project with the performance or(and) costs as you will get more users.

So the main lesson here is do not mistake the learning curve for bad tech stack. The former will bring you joy, and the later will just become a major headache and could even lead to project death.

What is the good choice definition then?

The good choice feels natural. Yes, I know how it sounds, but it is almost always like that. You might struggle at first while you understand some new concepts, but rather sooner than later, you should catch the flow. If it does not happen, try something else before jumping to the conclusion. Look for the following properties:

1. Your productivity grows as you learn the new framework
2. It does not take days to figure out how to do even simple things
3. You never debug the framework or technology itself - not like it could not happen, but it should not be the common case. Especially when you are sure that you want to do a simple thing (e.g., a REST API or stuff like that)
4. Main complexity comes from the business logic and not from the framework boilerplate/setup routine or what not
5. The framework passes the projected load test
6. In the case of open-source - solid and healthy community, working on the technology

Of course, this is a generalized definition, and it is often impossible to find a solution, checking all boxes, but the more you check, the more productive you will be long term.

Why fancy/popular technology is not necessarily a good choice?

At one point, an engineer might be tempted to think that just picking the most popular technology is always a good and safe choice. Unfortunately, this is not that simple. The thing is that we are living in the age of compounding innovation, where you typically have building blocks in the form of foundational technologies, which could be combined into something new or even innovative. So, in practice, such a truly generic foundational blocks are not appearing daily.

• A lot of technologies are simply not designed to be generic - they are designed to make your life easier in a specific use case. So, when jumping on some new tech, please make sure it solves the problem you have and not just have a lot of buzz around it
• Another class of technologies does not really solve a technological problem per definition but aims to improve your development experience via an extra abstraction layer. Sometimes it is helpful, but you have to always keep in mind that you will be limited by the abstraction layer's capabilities, somebody (with strong opinions) built for you, without knowing what you are doing. Additionally, you might end up building up team expertise around the framework and not around the technology itself, hence reducing the depth of your knowledge

For instance, I am thinking of something like MySQL being a foundational block, and ORM, being a thick abstraction on top of it. This abstraction is clearly powerful, but it is not a dogma, and you are free to choose something else if you feel that it does not meet your needs or that it brings too much complexity for your use case.

As an example of tech that is not fully generic, I would probably take serverless functions on your chosen platform. Clearly, the approach is powerful, but it could be that your workloads are simply not suitable for serverless computing.

If you just pick your tech stack based solely on popularity and amount of stars on GitHub, you might easily fall into the trap of a bad choice, which could badly damage your project.

The main advice I can give here - experiment a lot, especially beyond standard TODO list apps. Spend some time building a real prototype (based on your current project) and see how it feels, how productive you are, how readable the code is. Do not jump to conclusions too quickly. If, after a week of experimentation, it still does not feel right, get a second opinion, and eventually drop the approach. This is not the lost time - it is time you spend to build a solid foundation for your project.

So, is conservative tech always good?

Not anymore. Here is the thing. Users' expectations are growing crazy nowadays. Application is expected to be not only functional and useful but beautiful and slick as well. Your app is expected to work across multiple types of devices. Your storage is expected to be fast and durable. Your app is expected to have zero visible downtime. I can keep extending this list for a very long. Note the word "expected" here. Nobody forces you to do all these things; this is just the new minimal quality standard. If you do not follow it, your product adoption will be at risk.

Building the modern app without frameworks and abstractions will likely be a suicide mission in terms of code to be written and bugs to be fixed. But I typically dislike an approach when the dev team is starting to add too many new frameworks at the beginning, just because "we will need it in the future", or "look, here is a new cool thing". Like when you are just starting a new project and your dependencies list does not fit into your screen after a week of work. In my opinion, such an approach only adds noise and facilitates divergence from what matters for the project at the moment.

The way I would rather do it is:

1. Figure out what core technologies you are about to use and prove it to yourself, and then to every person in the team
2. Check if you can start like that, without introducing more complexity. And if you can start like that, please do so. If not, evaluate extra dependencies you need, as if it would be the core tech, and move forward

An example of the thinking process for an abstract REST service would be something like:

1. I need a database to store my data (single instance)
2. I need a web application framework to run my code on
3. I need a router to add new routes
4. I need a platform to run it all with one command
5. OK, Let's see if it is enough

So, in other words, brutally simple. No middleware, no multi-region DB cluster, no distributed tracing, etc. Such an approach allows you to start fast and evaluate your choices quickly. And later on, you might start adding more stuff to it, but only if you really need it. You can even change some of your core choices relatively quickly because your code will likely be written down to base principles already, which could be applied to any framework - even the latest one on the market with tons of fancy features. And you will do it, inevitably, but it will be in time and with a better understanding of the made a choice.

Where is the balance between two evils?

So it all boils down to finding the right balance: on the one hand, you do not want to bloat your code with tons of frameworks; on the other hand, you do not want to reinvent the wheel over and over again. You do not want to get behind in the current trends in the industry and what new tech has appeared within the last couple of months. My recommendations here would be:

1. Start simple - only use what is absolutely vital for your project, leaving any noise aside. Focus on what matters for your core foundation.
2. In the beginning, use frameworks and 3rd parties only for domain-specific code paths: like generic algorithms, maths, cryptography, etc. By starting like that, you will still have full control over your code so that you can adapt quickly
3. As soon as you've figured your core tech, start adding more stuff to it - 3rd party components that will allow you to enable a full modern UX and development experience. Note this part, you are still ending up with a lot of code from 3rd party vendors, but you are doing it a bit later in the product development lifecycle
4. If you are starting out a new primary project, think twice before using some shiny 6-month-old tech stack for it. I would rather test it first in some small component or a side project. Yes, it might sound a bit boring, but I am referring to professional software development here. So, we are optimizing for stability and productivity here, not for the level of hype. And if you will follow the previous three recommendations, it is very likely that it will be possible to migrate to that shiny stack later, as you will be confident that it's worth it
5. Keep experimenting all the time - this is not the lost time, but a solid way to keep up with the fast-changing industry and new concepts. In this case, you will be well equipped when making tech choices for your primary projects. You can also use super new tech for non-production systems at work, like various automation components, internal tooling, etc. It will allow you to get some instant in-house feedback before trying to integrate new tech into your main project.

Hope you enjoyed the reading! Want to discuss the content? Feel free to reach out to me on social media!