How to escape over-engineering trap?

Products and services are getting complex exponentially. Expectations are sky-high. In such an environment, an engineer is often tempted to build things "for the future growth", "to scale well", "to be orders of magnitudes better than existing solutions", etc. Though all those intentions are healthy and good, it is often redundant to focus on it, right from the start. The reality is that, it is highly unlikely, that you will be able to predict the future and build your tech stack in a way that it will remain unchanged for years. What is more important is to provide value timely, because nobody needs useless but scalable, perfect solution for the problem of the past

In this post I will try to cover a danger of over-engineering and more importantly, how to recognize and avoid it.

1. What is over-engineering?
2. Is your project over-engineered?
3. How to avoid over-engineering from the start?
4. Should we blame the product itself?
5. Can you save an over-engineered project?

What is over-engineering?

According to Wikipedia, over-engineering is is the act of designing a product to be more robust or have more features than often necessary for its intended use, or for a process to be unnecessarily complex or inefficient. And this definition covers quite a lot of this concept. However, I would amend it a little bit so that we add "at the moment" to the end of the former sentence. Look, technically speaking there is nothing wrong in building robust and versatile things. Moreover, you should always aim for it. However, one has also to accept the reality and map your engineering efforts to your reality.

Let's imagine you are a small company, building a brand-new collaboration platform. Are you sure your very first version should support 1M users and daily ingestion of petabytes of data? It is highly unlikely. Of course, you should design your system with scalability in mind, but to be honest, this is not so hard to do if you stick to basic building blocks in your tech stack, and not the latest framework that promises to fix all your problems at the same time. See this post to learn more about this idea.

Is your project over-engineered?

In my opinion, over-engineering is a type of the problem, which is easy to spot. Let's imagine, you received a pull request, adding a new and objectively trivial UI widget, or a basic CRUD API implementation. Now, I would do the following tests:

1. What is the amount of code, written in total? I know, it is bold and you need a little bit of experience to judge it, but seriously, is it like a 1K+ lines of code batch, or a compact and concise implementation? Trivial things must be trivial in any environment. Every added line is a technical debt. If you see that code volume is large, there are two possibilities: code is bad or framework does not allow an engineer to write a concise and dense logic. Both options are bad, for sure :)
2. What is ratio of code implementing business logic vs framework patterns, guidelines, logging and what not? Obviously, focus should be on business logic implementation and not some arbitrary chunks of code that have to be there because this is "the best practice". Of course, later in cycle you will formalize the implementation, add some metrics and stuff, but this should not be the main focus, initially.
3. How hard is it to make such a change in your codebase? This metric is a bit hard to quantify, but after 1-2 years of engineering you will be able to get the first estimation for common problems. Like, it should not take days, to add an API to the existing project (unless you are working at huge scale). And if it takes days, there are again two potential problems: either engineer is weak or code is too hard to change (for instance, too many places and very sparse logic).
4. How long is your dependencies list and how many of those are related to business logic? For any project, there is a tendency to grow its dependencies list over time, and this is ok. What's not ok is when you are working on initial project release and every request/state change is going through 10 middlewares(logging, profiling, crash reporting, resilience patterns, analytics...) just because it could be done. All these things are pricey in terms of cognitive load and, frankly, just scare contributors away, so fight for simplicity, like it's your last day.
5. Is every component/functions/class in your project claimed to be reusable and abstract? If yes, there are likely two problems: project is over-engineered and components are still not reusable for anybody except for the author. It is extremely difficult to build generic things, so my advice is: if you are doing product development, do not care about reusability too much. If one of the components will need to be reused, it will be a separate story and a separate refactoring. The difference is that in later case, you will at least know exact requirements and not make them up in your head.
6. Code is handling extreme edge cases and is full of legacy compatibility code. Sometimes all of that is required, but from my experience, at 70% of it is just a lack of engineering diligence. Specifically, it is highly likely that a lof of that code could be removed long time ago.

How to avoid over-engineering from the start?

For those who are reading my blog for a while, this is not a surprise - fight for simplicity and aim for logic density. If you follow this simple lead, you will never be wrong. In the worst case, your solution will be naive, but it is better to be naive, than pretend being extremely smart. Also, it is very natural to go from simple to hard and extremely difficult to go from hard to simple. I can give couple of actionable advices here:

1. Always focus on business logic first, because this is what your customers need.
2. Do not hesitate to oversimplify things. Sometimes, problems you are working on might be tricky, so do not afraid to assume that something exists already. Like, I will build part 1, assuming that part 2 exists already (even if you have no idea how to build that part 2).
3. Do not fall in love with frameworks, "best practices" and libraries. We talked about it here, but I will repeat, learn general engineering patterns and basic building blocks. Frameworks will help you to formalize things later, but please, start simple.
4. "Don't repeat yourself" (DRY) is not always a good idea. This concept advocates for writing reusable code, but remember, writing generic and reusable things is very hard and will come at the price of code complexity.
5. Validate fast and do not afraid to delete your work. If you are working on the hard problem, focus on quick validation pipeline. Build dead-simple prototype and see how it works... Do not polish it and later learn that you missed something, because you will be tempted to fix your code instead of deleting it and write it from scratch (which is often better, given that you just gained an extra knowledge of the problem domain).
6. If you are adding a framework to your project, please make sure you understand clearly what it does and why you need it. Note, I said "frameworks", because it is different with libraries, where you might add some domain specific dependency because you do not know that domain, but you know that your business logic needs it (math, 3d graphics, AI etc)

Should we blame the product itself?

Truth to be told, over-engineering could also be a result of a bad product management. Like if your roadmap fluctuates constantly and you have to pivot your product focus every month, it is hard to build something sustainable. Sometimes this is just a reality of a new startup. In this situation this is fine to accept and communicate, that whatever you are building is a prototype, which will have to be rebuilt as soon focus will be more clear. At least it will set right expectations and will not diverge your attention from the main thing: finding something that works for your (potential) customers.

Another situation to be in is when you are building something truly innovative in terms of technology. Imagine, you are building a Netflix in its early days. So, you are literally first to do it. And you know, it will be tough. In such case, it is also probably fine to accept that your first product version will be messy. But you will learn a lot, so that you can improve and simplify your solution later. It is important to keep it in mind and be ok with an inevitable fact that you will likely have to delete most of your initial code later.

So, to sum it up, sometimes the product itself pushes us towards over-engineering - it is particularly important to identify such situation, accept it and decide how to move on and re-iterate later.

Can you save an over-engineered project?

Frankly, this is hard. Let's say you just joined the company with an existing product. Couple of weeks later you see all the sins: legacy, code bloated with various framework(s) patterns, over-hyped team integrating every new hot library into the project every month, loads of dashboards so that it is hard to find something when you need it. You are all alone in the sea of sharks. It will be hard to convince people that some things should be done much simpler and that legacy code is eventually not required anymore.

My advice here would be: observe for a month or two, depending on project complexity. Could be you do not understand some things and it is just that the problem, project is solving, is tough. I saw people coming into teams and starting to advise how to build things from day 2 at the company. And believe me, this is tough situation to be in, if you are wrong. And if you will discover that project is indeed over-engineered, please make sure you convinced people that you are worth listening to. You can do so, by doing an excellent work even in the existing context. As soon you gained some trust, you can start looking around for trivial things that you can improve to make project code cleaner. Start simple. Believe me, if you will make a "Simplify whatever" pull request, removing 100 lines of code, without removing any features, it will be difficult to reject it. Simple is always better. Such approach will build up some reputation for you, so that you will be perceived as a pragmatic engineer, aiming for code excellence. Over time, people will start trusting your choices and you will be able to change more complex parts of the project. Just be prepared that it will take some time.

Hope you enjoyed the reading! Want to discuss the content? Feel free to reach out to me on social media!