Threadripper meets rustc

Improving build times the expensive way

I've been writing a lot of Rust code over the last few months and although there's lots of good work happening to improve the compilation speed, I often find myself starting a build then going to make a cup of tea or otherwise getting distracted.

Compiling

As an aside, one trick I often use when starting some long running command is to make it output a sound when it's finished. That way I can properly focus on something else without having to keep checking back to see if it's finished.

cargo build --release && \
	echo Back to work | espeak

Anyway I recently decided that I was drinking too much tea and therefore needed to buy a significantly faster computer.

I've been working on a Lenovo Thinkpad T480s and I think it's just about the perfect developer laptop. It's got a great keyboard (for my taste); it's light and silent; everything just works in Linux. However mine came with an Intel i5-8350U processor which isn't very ambitious and even then ends up being throttled fairly often.

I know you can buy a T480s with an i7 processor, and Lenovo have about 40 different variants of Thinkpads with increasingly ridiculous hardware, but you end up paying a huge premium for the portability and there's only so much they can do with something people are willing to balance on their knees.

So I ended up ordering an AMD Threadripper desktop. This is a huge jump from 4 cores to 32 cores and just feels stupidly fast. Also apparently Linus Torvalds has one.

I assembled my first PC over 20 years ago but I've become more risk averse in my middle age so I decided to order it from PC Specialist (no affiliation). Still slightly tempted to build it myself I priced up the components I'd picked individually and it came to about the same price as the build from PC Specialist including the 3 year warrantee. So yeah, happy to leave it to the experts.

Here are the important bits:

  • CPU: AMD Ryzen Threadripper 3970X 32 cores/64 threads (3.7GHz - 4.5GHz, 147MB cache)
  • Motherboard: Asus Prime TRX40-PRO
  • RAM: 128GB Corsair Vengeance DDR4 3200MHz (8 x 16GB)
  • HDD: 2x (RAID 0) 1TB Seagate Firecuda 520 Gen 4 PCIe NVMe (up to 5000MB/R, 4400MB/W)

Compilation performance

Here are some rough benchmarks to give you an idea of the performance gain, if you're wondering what a Threadripper could do you for you.

For each of the below projects I've done a standard fresh build having already fetched remote dependencies, submodules etc. so it's really only compilation and linking time.

I've included the Git hash of each repository so you can try out exactly the same build and play along at home.

I threw FFmpeg into the mix out of curiosity but the rest are all Rust projects although they might also be compiling some C/C++ under the hood.

_ Thinkpad Threadripper
Alacritty @ 5ee7ae8 82s 22s
rav1e @ 0fb9209b 104s 26s
Nannou @ 11ccd5b 183s 34s
FFmpeg @ 4e012fc8af 210s 23s
Servo @ a5a21a59ad 833s 194s
Rust @ 7d289aeade4 2,640s 213s

There's nothing very surprising here. The new system is vastly faster. Cargo/rustc does well at using all the cores to compile many crates / codegen units concurrently.

Building Rust itself with 64 threads

I suspect the final phase of compilation and linking is largely single threaded so there's a bit less of a performance gain than I'd expected for smaller projects and also for incremental builds.

Most of the time during development we're doing incremental builds so I think I'll actually see more benefit running tests than in compilation time.

One other scenario in which this will make a huge difference is when using git bisect to find the commit which introduced a bug. In that case we're jumping around the commit history, often changing dependency versions, and effectively having to do a fresh build & run tests each time.

Final thoughts

Aside from its performance and room-heating ability (the CPU alone has a TDP of 280W), the new system has made me think a bit more about how and where I work. At the moment it's sitting under my desk but it doesn't have a monitor plugged into it - I turn it on when I need it and use it via my laptop over the network. It could just as well be in a cupboard. Or in a data centre.

It's definitely worthwhile comparing the cost of ownership vs a cloud service where you can pay by the hour. For example a "c5.metal" EC2 instance with 96 vCPUs and 192 GB RAM costs $4 per hour at standard on-demand pricing, but it also has a 25Gbps connection to the EC2 network, which is handy.

I've actually been working somewhat like this for a certain client for security reasons. My entire development environment lives on one of their servers which I connect to over SSH. This particular arrangement is a bit extreme and prone to network latency & instability, but I could see it working fine if I could at least edit the code locally.

For now I prefer owning the machine and having full control over it but I can imagine the balance tipping in the future.