CyanogenMod has had explosive growth over the past year. When we started doing nightly builds, we only had to build for a few targets every night. We now have over 50 targets to build for, and I get requests about once a week to bring up new devices on the BuildBot. Many people wonder what they can do to help with this process, or ask how we are going to handle the new recommended build-system specs for Ice Cream Sandwich.
Background
We are currently using BuildBot to automate our nightly build process. This entire process involves for each target:
- Checking out the entire Android source tree, this is accelerated by using a tarball that is generated daily before the nightly builds run.
- Compiling the target from a clean state.
- Uploading the completed build to a staging server.
- Syncing the completed build across the mirror network.
- Adding the build to the get.cm index.
All of our nodes in the BuildBot cluster can accomplish this in under 18 minutes. However, the process usually takes a bit longer because the output of the build process is streamed to a central server and logged, which helps us identify build breaks. The current average automated build time is about 40 minutes per target. This can be greatly sped up by tweaking the process a bit, however it has not really been an issue thus far.
The ultra-nerdy, such as myself, will quickly realize that this works out to about 37 hours of wall time per day. We are able to complete the builds in less than 24 hours because we balance them across four servers, generously donated by EugeneKay, jeagoss and phaseburn.
The Future
Ice Cream Sandwich is coming, and the question of how we are going to handle nightly builds constantly comes up. Unfortunately source has not been released yet, so we can’t see how our existing hardware handles the considerably larger source tree. Fortunately JBQ has shed a bit of light on this, with a post on the android-building mailing list. JBQ is getting 25+ minute builds (wall time) on a pretty hefty dual-quad-core workstation with 24GB of RAM. This configuration is significantly more powerful than our current setup.
We won’t really know what we need until source is released, though some optimizations can likely significantly improve the chances of actually being able to complete all of our builds in one day.
How can you help?
Every time someone mentions a lack of resources in our build cluster, we get tons of offers to use idle hardware to do builds. This is great, and we’ve tried this before. Unfortunately this did not work very well for many reasons. The biggest reason was the hardware was not located in a datacenter, and there was no guarantee that it would be online and ready to start processing build requests. Next, when an update to the software powering these automated builds had to go out, it was very difficult to coordinate with the hardware owners to make these updates (most of them did not want to hand over root access, as they were running on their desktops).
Keeping the above in mind, we are still willing to take hardware donations, though the requirements are pretty strict:
- Minimum of 16GB of RAM.
- Minimum of 8 physical cores (hyper-threading preferred).
- Fast disks, SSDs are preferred, though SAS drives will do just fine.
- Server must be located in a datacenter, not your home (some exceptions are made).
- Minimum of 100Mbit Internet connection.
- Full root access required.
The server(s) should ideally be dedicated to building CyanogenMod, however it is fine if they are not. If you feel you meet the requirements above and would like to donate some servers, fire an email over to buildbot [at] ctso.me. If you have hardware, but no home for it, feel free to reach out as well. I’m sure we can work something out.