Whenever you are working on a programming project, chances are that there is some sort of tedious manual thing you are going to be doing in order to compile it or prep it for release. Compiling the project is usually not a huge pain in the ass, as it might have been the case a decade or two ago because these days we have smart IDE’s that will actually do most of the work for you. However, you are not always going to use an IDE, and the IDE you use can’t necessarily cover all the common use cases. In my experience, every project eventually requires a task that can’t be easily handled by the built in tools in your IDE. These use cases are commonly things like:
- Running a test suite the IDE authors did not like or approve of
- Deploying the code to a test server (which may involve cleaning up previous environment, copying files into specific directories, creating symlinks, etc…
- Creating custom installers with bundled third party dependencies.
- Running your code through custom quality insurance tools such as linters, optimizers, code signing tools and etc..
- Posting released binaries online, updating online release notes and etc…
A lot of this kind of stuff is finicky and weird and thus people will often do it manually just to make sure it is done right. In my opinion this is not the correct approach. If you want it done right once, then yes – maybe manual labor is warranted. If you want correctness on a consistent basis, the only way to ensure it is to automate the process. Human element introduces a lot of errors to a complex build process – it is far to easy for humans to forget things or miss tiny details. Automated process does everything the same way every time you run it, and it either produces valid results or fails in a predictable way.
I would argue that even simple weekend projects you might be hacking away at right now, would greatly benefit from a build tool. Especially if you plan releasing them on GitHub or some other place. If nothing else, providing a build script allows people who are interested in your tool/utility to reliably compile it on their machines even if they do not have much experience with the language and/or framework you used to create it.
I figured it would be interesting to see what do you guys use to build your projects? What is your go-to build tool? Is it Make? Ant? Rake? Something else?
There are two main compensating paradigms/philosophies in the build tool world. One states that build tools should provide the users with a minimum framework for running build tasks, handling dependencies and helper functions and then let them script their own tasks manually. Tools like Make and Rake use this approach by simply giving you a structured way to write your own build scripts. The other camp prefers to offer comprehensible collection of pre-made tasks that the user can just declare in their build file. So the argument is scripting vs. configuration. That’s the approach taken by tools like Ant or Maven.
Personally I’m in the scripting camp, because every project eventually needs a custom-made build task that will be a massive pain to implement in a declarative, config based build tool. I really like Rake for example, because I’d much rather write ruby scripts than fiddle around with XML config files. It is very powerful because of how flexible it is, and how easy it makes it to jump out into the shell in order to run external commands. It really is Make, but with the beautiful Ruby syntax and a library of helper classes that let you rapidly develop your build scripts.
Things like Ant, Phing and Maven have limited appeal to me because I just don’t think XML is the right language for this work. The terrible thing about XML as compared to say JSON is that it is needlessly complex. I always like to joke around that it is the configuration file standard that is neither human or machine readable. JSON and YAML files tend to be much easier to grok at a glance, and much easier to consume in your program. To parse XML config files you need a full-blown DOM parser which is non trivial.
That said, I do understand why it is used. I am aware of it’s complexity and robustness and the ability to audit and validate per existing spec are an important features that a lot of people want in their config files. I just don’t think config files are a way to go.
Grunt has a very interesting approach to this problem, as it more or less takes the middle road. Your average Gruntfil (a Makefile for grunt) is a JavaScript file that can contain build logic in a way similar to Rake. However, by default 90% of the stuff grunt file will be inside a bit JSON object. So most of the time, using Grunt feels like using one of these declarative configuration based tools that use built-in tasks or plugins to accomplish everything. The difference is that adding a custom task to Grunt is as easy as adding a function right in your Gruntfile. And you don’t have to smuggle it in by wrapping it in a CDATA block like you do in Phing – you just write it, then register it as a task and you are done.
So which build tool is your favorite and why?
I think SCons is really the best of both worlds, because there are some useful tools for it, such as the qt tool that can handle linking to the right files according to the dependencies of the program, but it is still a scripting setup like make. It uses python for the build scripts so they are easy to write, but can be made extremely powerful, since you have a full programming language at your disposal. Also, you can write your own python modules to do auto configuration for you, which can be nice if you have several similar projects that all need to do the same sort of build enviroment auto detection.
I like Make. For tons of things it works perfectly well, and it’s most of the time quite easy to understand at a glance because most of us know the terminal commands used.
For more complex Scala/Java stuff with dependency management and managed sources, SBT is pretty good though, and has a rather interesting approach.
On the other hand, I had to muck around Eclipse’s build process recently, and it’s not something I recommend anytime soon… Major headache ahead!
@ FX:
Also, Make is more or less the de-facto standard in the Unix/Linux world which makes it ubiquitous, mature and it’s peculiar quirks well documented.
I haven’t used Eclipse in a while, but I can imagine doing anything non-standard in it would be quite cumbersome. It worked well for compiling homework assignments when I was in school. On the other hand, at one point I had to install a few plugins to make it do something non-standard and it turned into a 3 day project where I had to downgrade my Eclipse to a previous version, and compile something from source just to get it to work, only to realize the performance was terrible and it would freeze up the UI all the time. :P
Coming from a Java background, I must say that Maven is a good basis until you need something more complex. As long as you can work alongside Maven, and do stuff their way, it is a great help.
Other build/dependency management systems, like Gradle, Buildr, Ivy, SBT and more use Maven’s way of handling dependencies – mostly because it works and that it covers all(most?) all open source software, and handles proprietary software through self-hosted repository installs (like Nexus and Archiva).
Maven also integrates nicely with (most) IDE’s, including Eclipse, IntelliJ IDEA and NetBeans, and other tools – including Hudson/Jenkins CI.
As said, it isn’t perfect, and you’ll probably grow out of it, but it works.
Either Make or Rake, much perfer using Rake but if anyone else may want to use the program then adding Ruby as a dependency (for non Ruby projects) is a bit much imho.
I have thorough experience with Make, Ant, Maven, Leiningen, and SCons. I don’t really like any of them that much, especially Maven.
Ant’s declarative configuration is appealing to me until, as you pointed out, I need to do something unusual. Then it’s impossible. However, I still think its biggest flaw is the lack of conventions: parts of it are still too low-level. Everyone has to invent their own conventions. If I’m building Java sources, this is my preferred build system.
Maven and Leiningen are painfully slow. Fortunately in Leiningen’s case I don’t need to build very often thanks to Clojure’s REPL. The only appealing thing about Maven is its well-curated repository, which I can access from Ant and Leiningen anyway. Leiningen is the only choice when it comes to Clojure.
SCons is painfully inefficient by re-running configuration checks on each build. The worst part is how it really fights you if you have an unusual environment, because it clears out all environmental variables for subprocesses. Have libraries or headers in a non-standard place? Too bad. I avoid using SCons.
Make can do virtually anything I need it to do; it’s my safety net fallback when everything else fails. But it’s too low level, making it really tedious and messy to maintain on medium to large projects (recursive makefiles, listing each individual file, etc.) and generally poor at portability. It’s also nearly impossible to write a non-trivial Makefile that isn’t broken in some way, whether for parallel builds or lack of proper argument escaping (a general unix shell flaw). It’s unusual to come across a Makefile that isn’t broken, including the ones generated by autoconf — try running autoconf in a directory containing a space, for example. Make is my build system of choice for everything else.
Still looking for that perfect build system.
@ Ron Collinson:
Very true. Then again we sort of live in a multi-environment age these days. Single language purism is a thing of the past in a way. We now have huge, language agnostic frameworks (JVM and .NET – both of which support a multitude of languages that all compile to cross compatible bytecodes) and projects that have cross-language dependencies. For example Yeoman has both Node and Ruby as dependencies (among a few other things).
@ Chris Wellons:
Heh, very true. Make is simple and straightforward all the way up until your project becomes complex enough to break it. :P
That’s the question of a sort “what’s better: an orange or a hockey stick?”
AFAIK, ANT is XML-based not because James Duncan Davidson liked curly syntax with tons of garbage. They successfully used some proprietary IDE with internal make-like engine. When Tomcat was opensourced they needed to create a build tool, which is understandable by likely infinite amount of unknown IDEs. Everyone who once tried to parse and represent an unstructured text in graphical manner (as most IDEs do) knows, that the only way not to come in the bump is to use highly-structured language. That’s why XML.
Furthermore, there is absolutely no problem to extend ANT/Maven with your own weird complicated task, written in high-level language (Java,) which is your fave since you are using ANT/Maven.
Ruby, on the other hand, is not a language you’re supposed to use within IDE. Neither it requires a huge repository of prebuilt tasks (basically, one may successfully spend a life coding in ruby, being on intimate terms with none of them, except, probably, RSpec/RDoc.)
Make was born when there were no goods around, except of “compile-link-me-this-in-this-order”. “No hooch, no gals, no nothing.”—as Raymond Chandler wrote about kinda same situation :-)
The summing up, I would definitely use Maven if my wife asked me to hack our android-based microwave; I do use Rake during my working days with rails and I like an elegance and simplicity of makefiles.
I would like an orange when I’m hungry and a hockey stick if I’m bored with these rake-maven-makefiles.
@ Alexei Matyushkin:
We already had s-exps. ANT had no excuse. http://harmful.cat-v.org/software/xml/
Personally, the only build tool I use nowadays is “go build”. Everything else tends to be interpreted.
Luke Maciak wrote:
XML configuration. Yuck.
The really nice thing about Make is that it’s trivial to just write a Makefile from scratch without even thinking about it, and without having to browse through heaps of documentation about obscure XML properties. Rake looks nice though, if you know Ruby.
@ Alexei Matyushkin:
Yes, but you can achieve about the same level of complexity using a format that is more readable. JSON and YAML are both good candidates.
Or you can do what Grunt does and make your configuration a JS/CoffeeScript file wrapped around a big JSON object for the declarative stuff.
XML is not necessary – it makes the declarative bits hard to read, but is not truing complete and thus won’t let you script in it.
@ IceBrain:
LOL, love that graphic. :)
@ FX:
Yeah, Makefiles are just shell scripts with some additional magic. Same goes for Rakefiles – they are ruby scripts plus magic.
Personally, I prefer CMake, mainly because I’m mostly working with C and C++. It has really good support for those and things that don’t fit can be hacked in with custom commands/targets. It abstracts away a lot of the low level specifics of whatever native build systems and compilers fairly well. Of course, there are some exceptions, but nothing you’d run into every day.
A top level CMake file could look like this for example:
https://github.com/peterix/MultiMC4/blob/master/src/CMakeLists.txt
And a bit of an oddball piece of java code could be compiled like this:
https://github.com/peterix/MultiMC4/blob/master/src/launcher/CMakeList s.txt
(note the other .cmake files in the same directory – those actually implement the java support)
Would I use it for something other than a C/C++ heavy project? Probably not :)