Summary (TLDR): Traffic on eBible.org and related sites overloaded my home data center. I rented another server in a real data center, subscribed to more Cloudflare services, and reconfigured things to handle the load and fix a confusing problem my old setup had. I learned that it is sometimes wise to not do everything as cheaply as possible. This also reinforced the idea that the most efficient Bible distribution is viral: letting people copy and share Bibles freely, as much as copyright restrictions allow.
I have been struggling for the last few weeks to keep all of my web sites online. I have 25 sites that I maintain, 15 of which are for direct Bible distribution, 2 are for Bible translators, 2 are for Bible-related software, 1 is this site I’m blogging on, 1 is for some old open source software Bible study apps still in use, and 4 are for related infrastructure. I thought that I had them all being served reliably enough from my home data center that I could just let them run unattended while I took a much-needed vacation. I was wrong—not about needing a vacation, but about my home data center being adequate to the task. So much for the vacation. What I didn’t know was that:
- The Cloudflare tunnel I was using to enable hosting on my home servers via either one of my Internet connections would cause infinite redirect loops for part of my sites from some ISPs. (I eliminated the cloudflared tunnel, since I couldn’t make it work right even with two AIs helping me.)
- Surges in demand would overwhelm my setup causing repeated outages. (I added another server and a load balancer.)
- My Eero home router would stop all traffic for about 15 minutes at a time more frequently than I thought for system updates. (This now only takes a few of the low-traffic sites down, so few people, if anyone will notice those updates, now.)
- AI bots would increasingly devour data on my sites to train themselves. (I could try to block them, but honestly, letting the bots at the Bible data will eventually benefit people who use the trained AIs for Bible research, so I just increased capacity to accomodate them instead.)
- Choice of web serever software actually makes more of a difference in performance than I thought. (I’m shifting from Apache2 to Nginx.)
- One of my home servers would randomly enter sleep mode after a reset. (I cured its narcolepsy with a systemctl command.)
- How vulnerable my home data center was to bad weather (i.e. hurricanes), even when they don’t hit head on. (Lola and Lowell didn’t actually take out the power grid or Internet at my house, but power and Internet outages were rampant near me. Overcoming this directly would take a large backup generator and/or an offsite server. For now, I opted for the offsite server.)
I went through many iterations of trying things. To make a long story short, I rented a dedicated server in a real data center for most of my web sites, and used Cloudflare to balance the eBible.org load among 4 different origin servers and cache the static files. The eBible.org site is highest traffic site I operate, so spreading its load out kept it from dragging down all of the other sites due to an overloaded server. Also, eBible.org should now stay online as long as any one of its four origin servers is healthy. When all four are healthy, then performance should be fast.
What I am doing now is scalable. I can tweak some more capacity out of the existing hardware with some software tweaks. If that isn’t enough, I can add more servers. But the biggest gain, now, is to encourage more people to share the Bible files they have downloaded. Imagine a candle light service where only one person tried to light all of the candles. Then imagine a normal candle light service where everyone who has a lit candle lights all neighbor’s candles. Obviously, the second scenario is much faster.
Digital Bible distribution is not free, but it can be free to the end user, thanks to our partners who support our ministry. If you are one of those, may God bless you abundantly!