The problem with over-engineered projects... our team inherited a project that was designed by some architects in an ivory tower and implemented by some other engineers.

It was quite complex (message queues, multi-threading, async etc.) so that it would be 'scalable' yet crushed under the slightest load, customers were experiencing delays during peak hours.

We removed about a quarter of the code (some intermittent queues & components), nothing changed in terms of functionality. Now we're removing more code and switching over to less complex data structures to fix the delays.

Maybe a complete rewrite would have been better after all, who knows.

I'm not a very experienced developer. I like to ship stuff and get projects done in reasonable time and a "YAGNI" attitude. Let's just rewrite stuff and introduce more abstractions when it actually hurts and a refactor is in scope.

On the other hand, I see a tendency that if you give projects (it could be a greenfield project or some maintenance) to a team of, say, 6 developers, then they will just 'create work' for themselves, filling up the todo list with items which may lead to an over-engineered project that could have been easily be done by 3 people within the same timeframe.

If you have some good reads/stories on this topic, patterns to watch for, I'd appreciate if you shared them.

> Let's just rewrite stuff and introduce more abstractions when it actually hurts and a refactor is in scope.

This is a very good attitude, but I would like to tack on a second part to it: Instead of designing your code to be infinitely extensible in all directions, make it easily replaceable instead.

It's impossible to anticipate every future requirement, but certain choices will make it easier to deal with a new requirement as it pops up. One of these things is unit tests. If you have some gnarly business logic with tons of edge cases, you really really want a good test suite because it enables you to add new edge cases with confidence.

I'm a strong proponent of this attitude. The most maintainable code is that you can easily delete.

It does require a lot more thought and discipline, because it's not just a case of constructing an epic hierarchy of abstractions, or stubbing/mocking the hell out of your code in the tests to swap things around.

Couldn't agree more. A little redundancy early on in a product's lifecycle, when requirements are still in their infancy, can be a very valuable thing. It's much easier to add a layer of abstraction later as requirements became clear, vs. remove an unneeded layer of abstraction (which is often impossible).

I've been on too many projects which tried to build the perfect abstraction right out of the box - as if the problem domain fully known.

I genuinely hate the software architect title. It's like when we were in school and the teacher told us to write an outline before writing. No problem ! the majority wrote the text first and then the outline.

That's because people in general write to think.* The level of deep thought achieved through writing is harder to achieve beforehand.

In the same sense, the level of deep thought achieved through writing code, is hard to achieve beforehand. It's actually worse because your engineers, dangled in this web of classes, won't be able to think about the big picture for themselves.

An Architect thinks in terms of what sounds good, what is beautiful in OOP-land not in terms of what is easy to write, and what's performant to implement.

One poignant example is the interviewer who wants you to implement chess pieces as classes. which sounds good in the architects' world but is blatantly insane if you think about the actual code and the actual challenges you're facing.

* writing to think and writing to be read are conflicting goals that why editors exist.

I was where you’re at 15 years ago; my language of choice was C++/Boost. I was advised to try C, not because it is a good language, but because it’s so hard to use, it helps you focus on the exact problem you need to solve. To this day I always “do one in C; do one in X”. I think Python (substitute any ‘batteries included’ language) can be good for this, too, as there’s no need to Yak shave with them.

Similar experience here. About 6 months ago I inherited a team of 30 that had spent 4 years building some monstrous microservice "thing" that didn't work and never seemed to build reliably and barely worked even if it could be deployed. Lots and lots of late nights trying to "deploy to testing" and wasted weekends restarting things.

I stopped the work, broke the team up (and let a few of them go), re-assigned the goals of the project to 3 people who, working half-time, from scratch, and in Python, rebuilt the entire thing in 3 months, delivered it and it worked (and continues to work). "Builds" are now an scp to a VM, and a restart of a flask application.

My wife recalls a similar experience where a complex application she worked on (and ran reliably on very little infrastructure and with an entire team of 4), and serviced thousands of simultaneous users, was replaced (when a new CTO came in) by a giant, consultant driven Java project that required approximately 6x the infrastructure, 3x the staff to keep it going, and at launch could only serve 9 to 12 users at once. It was a debacle. But sunk cost fallacy forced her company to throw millions of dollars of consultants at it until it achieved some minimal level of barely available service. She and her boss left around that time.

She heard that a couple years later the CTO was let go, the consultants fired and it was all replaced by a team of 5 rewriting it all again from scratch (in Java) but like you also without all the architecture barf dropped in from orbit, and immediately went back to servicing the thousands of users again.

I'm convinced that this is the fault of some kind of consultant industry that seems to infest enterprise software circles that is really only good at inventing ways of designing systems that require more of their services, but never seem to actually deliver working systems.

As someone who fell into a similar situation just last year, I can say that it helps a lot when you have previous work that had been done before you were involved. Perhaps the existing project doesn't show you how things should have been done, but it certainly shows you some of the challenges that were faced and what not to do about them so you don't fall into the same traps as they did.

So it's not always "those architecture astronaut guys used the company's resources to study their webscale fantasies and I came to save the day", but rather, "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight".

I agree with what you say here to a large degree. We were able to reuse things like external service connections, accounts, contracts, some of the requirements documentation and some of the already provisioned hardware, VMs and so on. Those things are not trivial to put together and easily saved us several months of work.

Where I think I part ways with you is with this "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight". One of the big lessons we've learned as an industry involved in software development is to start small, iterate often, get feedback from users. What I see time and again with these types of overblown space elevator projects is a kind of fundamental...immaturity -- the people who run these don't seem to understand how to achieve results with the minimum required to do it.

They've never started small and iterated to large. The projects they've worked on have all been so enormous, and have taken so long, that they have very few data points to draw lessons from. Each iteration and growth cycle in a mature project creates lots of information that can be reapplied elsewhere. But for people who've only ever grown up in large enterprise software projects, that's the only approach they know, and there's only a handful of lessons they've learned.

It's not only architecture astronauts, but the entire ecosystem that supports "enterprise" software engineering: vendors, consultants, scaled agile experts, design tools -- even university programs that churn out enterprise ready engineers. The appearance of these things, and how widespread they are in certain circles, demonstrates how those circles have regressed and tossed into the rubbish bin the lessons that we've learned. The penalty that's being paid is that these same lessons have to be continuously relearned again and again, but instead of pushing against outside models of how to do things (e.g. trying to adopt physical engineering approaches to software), we have to push against our own industry.

When you go against billion dollar enterprise software industries, you end up sounding like a heretic. It's really only when you get a long portfolio of success stories can you succeed. But that's very hard to get in today's climate where people spend their entire careers trying to kill ants with nuclear weapons dropped from the orbit of Mars.

This was very well put and definitely agreed with many of the sentiments you've put forth about bottom-up (get up and running with as little rpc as possible) vs sideways development (copy the entire netflix architecture and try to make it work)

I just didn't put the blame on the previous team, maybe I should have for sticking with it for two entire iterations, still not entirely convinced. I think they couldn't help it and the decisions were already made for them by large faceless corporations. I guess what's the point of calling themselves engineers if they don't feel like they are in a cockpit pressing lots of buttons right from the start?

By profession, I call myself a computer programmer, not an engineer or architect in order to remind myself that I should weigh problems by starting from fundamentals such as a state machine, an integer constraint system or a breadth first search, before setting up service discovery or training neural networks.

We had a similar debacle back during the dot-com boom. We were a Microsoft shop and we got a new CTO who despised our monoculture and demanded a shootout against Java. A large team of consultants and employees labored for almost 10 months to produce a system that scaled to .. 2 simultaneous users while running on the latest Compaq servers with tons of memory. Meanwhile our legacy VB6 app was serving 1600+ users on mid-tier hardware.