I'd like to thank Etsy for writing this article. Let's Encrypt's availability makes large deployments easier, but they're still not actually all that easy to pull off in many cases. It takes a lot of thought and work to scale securely.

We at Let's Encrypt talk to a lot of large providers about large deployments and there's really nothing more helpful than being able to refer engineering teams to detailed accounts of previous efforts.

These writeups really make a difference because they accelerate other large deployments. The sooner those other large deployments happen, the sooner many more people will experience a more secure and privacy-respecting Web.

This is something that's on our "very soon" roadmap for customer domains at FastMail as well, so we're likewise very pleased to see a writeup.

I've googled this subject a number of times over the years and this seems to be the most thorough answer. Thanks for providing it!

However, for a small startup (like mine), I think it's still a bit too much work to get this working. I know that there aren't any other "automated" options out there anyway. Perhaps one day. :)

I'll be writing a blog post in the coming week and publishing it on HN regarding how we do it; its a simpler method I think, though we aren't the scale of Etsy yet, but should be useful for smaller startups to adopt.

Thank you for this. Like the GP, I am a start-up and with my current workload, it too is a bit too much.

Looking forward to seeing your work. Will definitely aim to provide feedback too!

(upvoted)

It really isn't, I was able to implement something similar in a matter of about a month by myself at my current employer.

Would you be able to document how you did this, so that others like the GP and myself can benefit?

Thanks

It was just a matter of the same thing Etsy did, for the most part, minus Apache/PHP plus nginx/Python. I used the acme python module, and Celery for queuing. It's pretty engrained into my existing system that I don't think it'd be much use for me to explain it too heavily.

>To connect all this together, our proxy hosts periodically query CertService for a list of recently modified custom domains. Each host then 1) fetches the new certificates from CertService, 2) writes them to disk, 3) regenerates a config like the one above, and 4) does a graceful restart of Apache. These restarts are staggered across our proxy pool so all but one of the hosts is available and receiving requests from the load balancer at any give time (fingers crossed).

How many pools are there? If you're having to restart apache every time you renew a cert, are they renewing certs in the background and then only restart apache when they have around a thousand renewed? If so, wouldn't the current running certs be revoked in that time? It seems like on-the-fly cert usage is something that could be made, if not already.

I can't speak for Apache configurations, but on-the-fly cert lookup is possible in nginx using the lua_nginx module[1] - IIRC that feature was developed by Cloudflare, to power their SSL termination.

[1]: https://github.com/openresty/lua-nginx-module/#ssl_certifica...

I was also wondering about this. I would have put in the engineering effort to migrate to nginx (at least as a front-end proxy) from Apache for this. Seems like a lot of reloads that could be avoided (even doing graceful reloads in Apache gives me the willies).

LVS/IPVS is OK, but what does it have to do with SSL termination and certificate churn?

I don't believe LE revokes certificates that get renewed, but rather lets them expire. If they implemented their stuff correctly, it should renew roughly 30 days before they expire, which in turns allows for plenty of leeway in adding in new certificates.

Was just talking with my boss a few weeks ago regarding how we might allow customers to use a custom domain name in the future. Was dreading that implementation but now it's looking a little brighter with this writeup. Thanks Etsy!

It's a good article, but it's a bit light on some of the technical points. We tried doing something similar about a decade ago (before the widespread adoption of SNI) and ran into the classic connected name/host header mismatch problem. Here, it's not explained that client has to have a way to suggest which certificate it's going to validate, and that there wasn't always an answer for that problem.

Also curious to know more about their LB solution and how it scales. Encrypting everything isn't free if you're doing thousands of ECDH[E] handshakes per second to chase down that "A+" rating.

You can basically assume SNI support on all clients. Etsy is using it based on that fragment of Apache config.

Is't only me, or their initial reasoning ('it's pricey' and a screeenshot of a certificate being $1500+ when you can have it for less than $10) moves the focus of why Let's Encrypt it important - not that it's free, but that the ACME protocol gives you the freedom to generate certificates very easy, and have that setup process even easier for multiple (think all) services flawless.

Congrats for them for supporting Let's Encrypt as an organization (it's not mentioned in the article it is in the comments here).

You're right, but i'm afraid to imagine how much it costs to get an enterprise account with one of the major CAs with API access to issue hundreds of thousands of certificates.

Where's the part where they donate to Let's Encrypt as a thank-you for making all this possible?

Head of Let's Encrypt here. They did, but did not mention it in the article.

Great to hear, I was thinking the same thing.