Wednesday, September 9, 2015

CalTopo Goes Commercial

I've spent a lot of time lately thinking about the future of CalTopo.  While I'd love to claim that I started with some kind of genius original vision, the truth is that it started as an offshoot of a hobby project I'd been working on for SAR, and grew organically from there.  As with any software project, the work started out fun and gradually became more of a drag.  Now much of my time is spent answering emails, fixing bugs, and dealing with operational issues.

In short, the first 10 hours or so a week I spend on CalTopo have all the excitement of a regular job, but without the upside of getting paid.  I only get to do the fun stuff after I eat my broccoli, and many weeks that never happens.

Happier times, when CalTopo's landing page didn't have a map viewer, you had to create a map before you could draw anything, and I didn't have any support work.

After devoting my free time almost entirely to CalTopo for a number of years, I've been getting restless.  The past year saw me delve into researching the impact of terrain on lost person behavior in SAR.  While the results may seem obvious, an entire generation of search managers has been trained using the aeronautical/maritime model of gridded area searching with little regard to terrain.  The Department of Homeland Security, which fancies itself as owning inland SAR in the US, is still heavily invested in that model, as is the current crop of textbooks.  My research project is a tiny drop in the sea of effort required to restore some sanity to the way people are found, and I would like to do more.

Another consideration is my AWS hosting bill.  August cost me $2200; rounding a bit, the costs were roughly:

  • $250 to host map tiles on S3
  • $200 to host the and servers
  • $100 of random bits and pieces, such as EBS snapshots full of GIS data that I need to keep around
  • $1400 in data charges, mostly serving map tiles


There are a few smartphone apps that license the map tiles from me; they form the bulk of the $1400 in usage costs, and also pay me just enough to subsidize the entire AWS bill.  However if they were to pull out I'd be up a financial creek: beyond the fixed hosting costs, CalTopo users alone account for $100 a month worth of map data.  This is not some forum that can be run on a cheap shared host, and while I greatly appreciate the various donations people have made, they fall well short of the site's actual monthly operating costs.

The costs involved with developing and running CalTopo - both real and opportunity, financial and time wise - have been significant.  While I still cry a little inside every time I price out a 4x4 Sprinter camper van, I don't regret the hours and funds plowed into it.  However, looking to the future, how long can I continue channeling such a large part of my life towards CalTopo?

Yeah, Baby!  If I could go back in time, would I trade CalTopo for one of these bad boys in my driveway?  Decisions, decisions.

I don't know what the answer is, but it's certainly not another 5 years.  Even ignoring the pull of other projects, I've simply been spending too much time locked away behind a desk.  My own sanity requires more weekends spent recreating rather than working.

As I see it, there are basically three options:

Keep the current model and let the site stagnate.  I'll occasionally be able to make time for improvements, but the pace will slow and large features like auto-routing along trails are unlikely to get implemented.  If map licensing fees drop off, either shut down or eat the $7k/yr in unavoidable hosting costs.

Open source the code and start a nonprofit to cover costs.  I tried the open source approach with early versions of the code, and no one ever contributed anything (literally, which is why I was able to return to a closed source approach).  With the site's improved popularity it may be a different story now, but I'm not convinced that the open source approach works well with hosted sites, as opposed to software that end-users run locally, or server frameworks like Apache.

In addition to code improvements, there are also the hosting costs to worry about.  I've seen plenty of nonprofits struggle for cash; the most effective ones either have strong local appeal or some kind of national advocacy message.  A backcountry mapping / SAR nonprofit wold have neither, and it's unlikely that I would be able to do more than struggle to cover each year's hosting costs.  That said, if anyone has a pile of money and a burning desire to help the outdoor and SAR communities with it, drop me a line at

Get serious about running the site like a business.  Beyond hosting costs, try to make enough to justify continued development and investment into the site.

After a lot of thought, I've decided to go with option 3.  While there was some hesitation leading up to the decision, now that it's made, I'm pretty excited about it.  A sharper focus on commercially viable features will ultimately mean a better site for users, with me forced to spend more time on documentation and UI improvements rather than chasing shiny new ideas.  It opens up the potential to include paid map layers, such as PDFs with higher resolution aerial imagery.  And, hopefully, it will allow me to invest more hours developing new features.

The "shared maps" layer was originally driven by commercial ambitions, but now that it exists, I think it's a great feature in its own right.

Understandably, I don't expect all the users out there to share my enthusiasm.  Since I'm explaining why I need to make CalTopo commercially viable, I also want to clarify a few things:

  • CalTopo will continue to be usable as a free site; it will not be hidden wholesale behind a paywall.  As a user of the site myself, I want to be able to bring a map up on a friend's computer without having to sign in.
  • CalTopo and SARTopo will continue to be free to first responders who use them professionally, such as SAR and wildland fire.
  • Billing will be simple and transparent.  You won't have to enter a card number to start a free trial, and  won't have to time your cancelation with the end of a billing period.  You can cancel with a single click, no annoying retention process.
  • You can keep your data.  You don't have to worry about losing any of your saved maps or PDFs if you cancel your subscription.
  • If you've already made a donation, you can request to have that credited towards your account.

I'm still working out the exact details and will lay them out in another post.  However, my current thinking is a $40/year subscription that includes the following:
  • Unlimited private maps.  Free users will be limited to 5 private maps, although they can create as many publicly shared ones as they want.  If you already have more than 5 private maps, you can keep them, you just can't save any new ones until you get the total under 5.
  • Save PDF URLs for up to a year.  For free accounts, URLs will expire after 30 days.
  • Large-format PDFs.  Free accounts will be limited to 11x17.
Edit: I should have made it clearer that this is simply one option.  I'd also consider something that's cheaper but encompasses more of the site; see comment below.

One feature I'm still contemplating is a smaller cap on the number of pages a PDF can have (current limit is 15 across the board).  There may also be some additional paid PDF management functions, such as the ability to hide the QR code or point it at a custom URL.

So . . . what say ye?  What's the best price / set of features that will give me a functional free site and happy paying customers?

Friday, August 28, 2015

Up Again

At 8pm on Tuesday, I got paged out for a typically inconvenient mutual aid SAR: pack everything I need for a 72-hour deployment, sleep a couple hours, wake up at 3:45, drive 6 hours, fly into the middle of the Sierra backcountry, and hike out 40 miles.

Thanks for the lift, SEKI H-552

When I get back to civilization, my phone normally goes crazy with follow-on SAR pages.  This time, there was also no shortage of emails from worried CalTopo users.  Some people saw this:

some couldn't connect to the site at all:

some couldn't print maps, and a number sent me worried emails stating that they no longer had any maps saved to their account.

While I normally find someone to keep an eye on CalTopo for me when I leave on long personal trips, there was no time to do so before this search, and the site was down for at least a day while I was off the grid.

What Happened?

Pretty simple: the CalTopo server ran out of disk space.  This is mostly due to the tile cache - PDF generation, the view from here feature and various layers require me to fetch tiles from external sources, do some work on the server, and then send them off to the client.  Rather than fetching a map tile each time it's needed, I cache it locally on the disk.

In theory, the cache is supposed to have a maximum size and purge tiles to prevent itself from growing too large.  In reality, this isn't happening.  Since the cache gets reset every time I push a code change, this has generally not been an issue.  However with CalTopo growing, and the number of tile requests increasing proportionately, I've apparently hit the point where it will fill up in only a couple days.

Easy enough to fix, at least in the short term.

What About My Data?

The lack of space caused database connection issues, which meant that CalTopo couldn't pull back maps linked to individual accounts.  The maps were never actually gone, and I didn't have to do anything like restoring from a backup.

If something catastrophic were to happen, I back the database up to Amazon S3 nightly, so - again in theory - I should never lose more than a day's worth of changes.  I do occasionally download the nightly backups to a local drive in case my Amazon account credentials get compromised, but that would mean all kinds of problems.

If you're still concerned about data preservation, you can back up maps locally by choosing "Download CalTopo Backup" under the Export menu.  This is also a way to copy one map's contents into another, while preserving line styles, comments and so forth.

Tuesday, August 18, 2015

Please Share Your Maps

Since the beginning, CalTopo has had the ability to make maps publicly visible, i.e. someone can find the map without knowing its URL.  However all this did was add the map to an obscure page that few visited other than Google.

Today that changes with the introduction of the shared maps layer.  It's available as an overlay checkbox, along with a layer for your own maps that's visible when you sign in:

At wider zooms, each shared map is shown as a simple bounding box and center marker.  This can get a little confusing - some additional UI love may be necessary - but mousing over highlights the box and center mark.  Clicking brings up an info window with a link to that map:

Zoom in a little further and individual features get displayed:

Mouse over a feature and everything from the source map will get highlighted, giving you an idea of what that map covers.  Clicking gives you the feature's name and comments, if any, and another link back to the source map.

So how do you share your maps with other CalTopo users?  When saving a new map, just choose "publicly visible" under the "Share Your Map" section of the save dialog.  Go ahead and fill in some details in the "Tell Us About Your Map" section as well.

For maps you already have, start by clicking on your account name at the top of of the left bar.

Then go ahead and and set the sharing dropdown to "Publicly Visible" for each map you want to share.

This might be a good opportunity to visit the "Your Account" tab in the same dialog and change how your account is displayed to other users.  I'm also putting a messaging system in place so that other CalTopo users can ask questions about your maps.  If you don't want them to be able to contact you, uncheck that box.

There are only around 500 maps currently shared on CalTopo.  As that number grows I may need to change some of the implementation details to improve performance or reduce information density, but I think the basic approach is here to stay.  I'm pretty excited about it - I've already discovered a few maps that I wish I'd known about sooner.

Thursday, April 2, 2015

Improved and Printable Elevation Profiles

Elevation profiles are one of the few remaining gaps people have noticed when comparing CalTopo to desktop software.  No more.

There are now two profile modes - the existing interactive one that uses Google elevation data, and a full-page version that builds on the terrain statistics dialog with CalTopo sourced elevation data.  You can get to the new profiles using the terrain statistics menu option, or clicking the expand link at the top right of the interactive profile dialog:

The expanded version is a full-page dialog:

One of the most obvious differences is axes in place of rise/run data:

There's also a heatmap below the profile showing slope angle, land cover and tree cover along the route:

Below that is the previously existing terrain stats content (lines only - terrain stats along a polygon's perimeter, to match the profile, would be meaningless or misleading).  A more subtle addition is that nearby markers are automatically placed along the profile:

Using bulk ops, you can also plot multiple lines on the same axes:

Absolute plots the lines' actual elevations, while relative shows how each line changes from its starting elevation.  Below are 3 routes plotted against each other in absolute mode (top) and relative (bottom).

PDF versions of any profile are available using the print link at the top right of the dialog.

Note that because the interactive profile uses Google data and the expanded profile uses CalTopo data, gross gain and loss numbers may be slightly different.  Like terrain stats, expanded elevation profiles will only work within the continental US.

Wednesday, March 25, 2015

Weather Stations

CalTopo now has a source for NOAA weather stations - some are in standard airport and urban locations, but a number of others are in quasi-backcountry locations useful for trip planning.  Note that the data is screen-scraped off of a web page rather than coming through a documented web service, so it may be less reliable than the SNOTel and water gauge layers.

available as a data overlay checkbox

Similar to the other data layers, at broad zoom levels only station locations are displayed, not station data.  Zooming in, stations with winds greater than 10mph are rendered with arrows showing the wind direction, while stations with light wind or no wind data are shown as black circles.  Mousing over a station brings up a tooltip with the last observation time:

As with SNOTel and stream gauges, clicking on a station brings up additional details and a link to the station's official observation pages.  The official pages contain additional information I'm not sourcing, like solar radiation, fuel moisture and cloud ceiling.

The details dialog also has links to government-sourced diagrams showing typical monthly daytime and nighttime wind patterns.

Between weather stations, SNOTel and stream gauges, CalTopo now provides hourly data from over 12,000 remote sensors nationwide!  As usual, please report any issues.

Friday, March 6, 2015

Killer Visuals Through Enhanced Relief

One of CalTopo's core features has always been the ability to blend multiple map layers together.  Until now, that's been done by creating a layer stack and varying the opacity of each layer to suit.  This works, but shaded relief has always been a weak point.  In order to keep the relief layer from washing out the rest of the map, I had to crank up the exaggeration.  This meant you could use less relief and get less washout, but it reduced the layer's ability to capture small details.

Even so, it's never played well with aerial imagery.  Consider the image of Mt Shasta below; you can make out terrain features from the relief layer, but the imagery itself is starting to get washed out.

Professional mapmakers solve this problem by using more advanced techniques than opacity based layer stacking.  One of those is to do a multiply blend, where each pixel on the map is multiplied by the intensity of the hillshade, on a 0-1 scale.  This darkens the image a bit, but does a much better job of preserving it.  The photo below is the same image of Shasta with multiply blend relief shading . . .

There's no way to a multiply blend in Google Maps, so I have to fetch all the tiles on my server, combine them into a single image, and then pass them back to the browser.  This means that when using the enhanced relief layer, the map won't react to react to opacity adjustments in real time - it has to fetch a new batch of images instead of just changing a CSS parameter.  It also means that images not available to my server - like Google's - will default to the standard relief shading.

While the effect is more subtle with maps than with images, but I still think it's an improvement.  Below are maps of Mt Shasta with both standard and enhanced shading:

This is still experimental, so I may make some changes going forward, including switching to an approach that lightens some parts of the image and darkens others.  While enhanced relief works with printing, it reverts to standard shading for KMZ exports.  Still, I'd rather get it out there and get feedback than tinker away privately until it's perfect.

Wednesday, February 25, 2015

Dude, where's my UI?

CalTopo just gained a redesigned user interface.  Normally I hate invasive UI rewrites, and I'm sure this one won't please everyone.  However, over time it's become apparent that CalTopo's previous long-standing UI had too many obscure and confusing features.  It was painful watching people struggle with the same issues over and over, and while I may not have solved all of them, I felt that I had to at least try.

The most obvious change is a transition to pull-down menus across the top of the screen, grouping features from the left bar, top right and context menu into a single location:

Some of the context menu options proved particularly hard for people to find, like view from here and terrain analysis,.  While they still exist in the context menu, they've also been pushed out to a Measure pulldown menu:

Layer selection is largely unchanged, but has been moved into a more obvious standalone control that mimics the de facto standard look for a number of programs these days:

Mousing over the control, layer addition and removal should look familiar:

One big change is that clicking on a marker or shape will now bring up an info window with distance (lines), coordinates (points), and several edit and analysis options.  Clicking an object in the left bar still centers the map on that object, but will also bring up the info window:

Account and map management options in the left bar have also been consolidated and should hopefully be a little more obvious.

The "Share this map" option now only gives you the URL for sharing, while "Manage this map" lets you set both general map information and read/write access for other users.  The account link leads to an updated and hopefully more straightforward one-stop account dialog.

There are numerous small changes as well - as just one example that's hard to screenshot, the new marker dialog is now accompanied by a draggable on-screen marker rather than being in a fixed location.  As you drag the marker the coordinates dialog will update, and vice versa.

While I've done my best to test the new code, I'm not delusional enough to think it's error free.  If something looks off, let me know.