At 8pm on Tuesday, I got paged out for a typically inconvenient mutual aid SAR: pack everything I need for a 72-hour deployment, sleep a couple hours, wake up at 3:45, drive 6 hours, fly into the middle of the Sierra backcountry, and hike out 40 miles.
|Thanks for the lift, SEKI H-552|
When I get back to civilization, my phone normally goes crazy with follow-on SAR pages. This time, there was also no shortage of emails from worried CalTopo users. Some people saw this:
some couldn't connect to the site at all:
some couldn't print maps, and a number sent me worried emails stating that they no longer had any maps saved to their account.
While I normally find someone to keep an eye on CalTopo for me when I leave on long personal trips, there was no time to do so before this search, and the site was down for at least a day while I was off the grid.
Pretty simple: the CalTopo server ran out of disk space. This is mostly due to the tile cache - PDF generation, the view from here feature and various layers require me to fetch tiles from external sources, do some work on the server, and then send them off to the client. Rather than fetching a map tile each time it's needed, I cache it locally on the disk.
In theory, the cache is supposed to have a maximum size and purge tiles to prevent itself from growing too large. In reality, this isn't happening. Since the cache gets reset every time I push a code change, this has generally not been an issue. However with CalTopo growing, and the number of tile requests increasing proportionately, I've apparently hit the point where it will fill up in only a couple days.
Easy enough to fix, at least in the short term.
What About My Data?
The lack of space caused database connection issues, which meant that CalTopo couldn't pull back maps linked to individual accounts. The maps were never actually gone, and I didn't have to do anything like restoring from a backup.
If something catastrophic were to happen, I back the database up to Amazon S3 nightly, so - again in theory - I should never lose more than a day's worth of changes. I do occasionally download the nightly backups to a local drive in case my Amazon account credentials get compromised, but that would mean all kinds of problems.
If you're still concerned about data preservation, you can back up maps locally by choosing "Download CalTopo Backup" under the Export menu. This is also a way to copy one map's contents into another, while preserving line styles, comments and so forth.