Let me know if this has ever happened to you. Your boss walks up to you and tells you he has a tiny little, itsy, bitsy project for you. He wants you to build a small online application. Nay, a small online form – just a single form with some database back end. Nothing more. A single page that would allow the employees could submit their TPS reports (or whatever) online. It’s only going to run on the intranet, it will never face the web and there is only 4 people who will really be using it so there is no point worrying about authentication, user management and all that stuff – just do the bare bones minimum necessary to tell the users apart, and keep them from overwriting each others data.
You implement it, and everyone loves it. The boss pats himself on the back for clever use of technology and tells you to add a tiny little feature to make the thing accept quarterly evaluations as well, and extend the user base to like 20 people. That’s about it. And you shouldn’t really worry about expanding it. It’s not going to get any bigger than this. It’s just a tiny change. No need to add anything else. Don’t spend any time improving the design. It’s not necessary, and you will be wasting time.
Next week you get another request. Then another. And another. Each time it is a tiny little change – and you are explicitly told the application is not going to grow, and it does not need a redesign. Six months down the road your application is the main online hub of the company. It’s facing the web, it’s tracking just about every little bit of data your boss could think off, stores few gigs of data and every employee, client, applicant and visitor must interact with it at one point or another. It is a monster and the code is a labyrinthine maze of hacks, patches workarounds and hastily added modifications aiming at adding stability and security to something that was not designed for it in the first place. And better yet – no one, including you can believe how huge it got and how quickly it happened. No one has seen it coming. No one could predict it would actually be taken this far.
I’ve seen this happen multiple times, and heard about similar scenarios from others. Very small, simple projects have a tendency to blow up and grow exponentially into huge enterprise systems. You never know which project is going to bloat this way. In fact, you probably won’t know that one of your projects is on this destructive path until it’s to late. It happens in small incremental steps, spread over long amount of time. But it happens.
And since we know it happens, we can be prepared for it. The easiest way to avoid exponential growth of this type from becoming an issue is to always code as if you were designing something 5 times as big. Always modularize your code, build your applications in a classic 3 tier system and always use MVC or similar paradigm. If your app bloats into some sort of a monster, you will have the infrastructure in place to support it, and build it up. At least for the most part.
Then again, this approach flies in the face of the KISS principle. Sometimes coding this way is indeed an overkill. Sometimes a single form will just be a single form and building a 3 tier architecture and creating/deploying some sort of a framework to support it may be a huge waste of time and resources. Sometimes a quick, dirty and direct approach can be vastly superior to the roundabout enterprise way.
So I guess the trick here is to do something in between these two extremes:
- Always assume your code is going to be facing the real internet, even if the initial spec says it wont. Build with security in mind.
- Always assume you will need multiple access levels for your users and a flexible access controls.
- Never assume your data won’t bloat out of proportion. Never use sqlite or SQL Server express when you could be using something more scalable
- Design your database for extensibility – normalize your design and be aware that you might be adding more tables and more complex relationships into the mix in the future. This shouldn’t be much effort since your tiny project will probably need only a handful of tables.
- Try to do some data modeling and object-relational mapping as this will help scaling the code later. Since you are starting small, this should be easy to do and it will keep your code clean and organized
- Design or steal a robust log in / user authentication module. One day your application may become the main login to the intranet or myriad of tacked on services. Don’t half-ass it. Also think how you can handle authentication from robots – since you may need to set up complex web services one day.
- Use an off the shelf module or solution if possible. Why? Chances are the author already spent a considerable amount of effort to ensure extensibility and scalability of their code. So when that inevitable feature request comes in, you have that much less work to do. And even if it is not very extensible, you may still be ahead. It is much easier to justify the need to do some re-writing or re-designing when you can blame it on inferior 3rd party code.
Feel free to add your own tips to this list. Did this sort of thing ever happen to you? I speak from experience here – I still maintain a system that started like that. I was young, naive and green undergrad and I got a tinsy little web project. And now here I am, many, many moons later – still maintaining the damn thing. It grew into a behemoth – an overwhelming mountain of code some of which is so crappy that I am the only brave soul that dares to touch it.
I also inherited a system like that once. I rewrote most of the UI (that was actually my assignment) and left most of the back end intact – trying to avoid the hairy code on that side of the application. I then happily passed it on to the next brave soul.
How about you?
Happens all the time. I am currently developinig with BMC’s Remedy AR Server, building custom help desk, ITIL-compliant stuff. More often than not, we have to look at what something MIGHT bloat into. For instance, one or two Statuses on an Order might have corresponding Substatused (a Status of Pending might have several different Substatuses). You could hardcode it so that the Substatus field becomes writable if the Status is set to Pending. Simple to code. But if the Client keeps requesting new or modified Statuses, and adding Substatuses to them, then you have a situation in which you keep adding hardcode for each Status that has a substatus.
The best approach is to table drive the Statuses and have related Substatuses (of course), but most people don’t think they need to data drive something finite like a list of Statuses.
You nailed it. That’s exactly what I have seen happen with several projects at my current workplace and my last one. A small tool built for a single, simple task incrementally grows to become a big tool that not very good at a bunch of things. It also generates a growing list of issues, a list that never seems to be able to shorten despite any time it spent fixing issues.
Early modularization and security are probably the most important things. Hacking on security later won’t be very secure.
To add a tip: if it seems your project is growing like this, make sure you support it with unit testing (or some kind of testing) right away. Don’t put it off! Tests will make you a lot more confident in rewriting/refactoring code to be more maintainable. I would expect confidence to be proportional to how often and how quickly you rewrite/refactor.
Good post, this has happened to me more times than I can count. “It is just a small VBA macro” becomes thousands of lines of unmaintainable kludge after kludge. What we started doing was this – every time we underwent a project we filled out a form. The form basically defined scope of the project, time allotted for completion, resources to be used, and a general spec. Changes to the scope needed a new form, new time allotted, and sometimes required new resources & spec. It was more overhead but it stopped the issues – small projects stayed small, large projects were build right from the start.
@Steve: Yeah, I’ve seen this sort of thing once. And as you said, the best approach here is to put these statuses in the database and then generate the pulldown menus dynamically – when the main one changes, just re-build the sub-status one from the DB and you are set.
@Chris Wellons: Good tip – building unit tests as you build the application definitely helps.
@jambarama: Ooh! Good idea with the spec document. It formalizes the whole process. :)
At my daytime job the main project is very similar to what you described. First it started as a medium size system with restricted functionality – sort of, this is what we got, take it or leave it. The client base grew, some large clients appeared and started to ask for additional functionality. And at some point the decision was not to make a new version of the product but to make patches and fixes for the certain clients. Then other clients learned about the additional functionality and asked for it too.
Now, 7+ years later it’s a code monster – it hasn’t been documented well (some things haven’t been documented at all), nobody knows its full capabilities, it is all hacks and tricks and hardcode. And guess what, there were at least two efforts to rewrite the damn thing on new platform (the old one is ColdFusion 5) – first .NET, then Java. Both were buried – the estimates were two years of development. I won’t even use the word ‘architecture’ near that thing – it’s crutches supporting crutches. Now there’s an ongoing project to move to ColdFusion 8 – and the preliminary results are that it would break in so many places….
Our biggest client decided they were up to recreating the functionality themselves. 1.5 years and they weren’t even close. So they bought us :)
The interface is… it feels insulting to other interfaces to call it that. No two pages are alike. It’s all frames and nested tables and if you try it with Opera it will ask you to download IE5 or newer. It falls to pieces in IE8 and our latest project was to add EmulateIE7 meta tag.
Sorry for the extensive whining :)
@Victoria: No worries! Oh, and you wouldn’t imagine how common this is.
Just the other day I talked with a guy who told me he used to be a COBOL programmer back in the day. He recently was able to visit the company where he got his very first job out of college. Apparently, the COBOL application he helped to develop is still in use right now. Apparently they have tried porting it to ASP few years ago but failed miserably. They are currently working on creating an ASP.NET port.
Funny thing is that the port – written in modern programming language, using best practices and running on very expensive new servers runs much slower than the old application which is a labyrinth of hacks and ugly kludges running on 15 year old hardware.
I don’t understand 3, why don’t we use database abstraction libraries, so we can move from SQLite to a “big” engine without much hassle? I’ve never dealt with huge databases, but unless you need the database specific functionalities it should be easy to switch when needed.
@IceBrain: Well, what I was getting at is that migrating to another DB system will always be a hassle. Even if you do the right thing and use database abstraction libraries you will still have to worry about moving data over to the new system.
If you are lucky, you can just make a SQL dump of data on one end, and import it on the other – but that doesn’t always work since different DB systems often like to use slightly different SQL syntax.
So you either sit there and try to clean the SQL up until it becomes system agnostic enough to do an import or you start looking at migration tools which may or may not be free.
If you are lucky – it can be a walk in the park. If you are not, it can be a huge hassle. Either way it can be avoided if you pick MySQL or Postgre instead of SQL Express which constrains you to 2GB of data forcing you to switch to something else as soon as you start growing.