Sunday, December 6, 2015

PDF Improvements

CalTopo PDFs are now sporting a couple of improvements.

The most noticeable of those is a title input field on the print page:

If filled out, it will show up in a couple places.  On the bottom of the PDF itself:

on the account dialog:

and also in Avenza PDF Maps:

Slightly less noticeable is an improvement in label positioning.  Before, labels were drawn without regard to each other; on a busy map, this resulted in an unreadable mess.  Now, labels will not draw on top of each other.  If two labels conflict, the PDF generator will attempt to move them around (to only few positions; it's not very smart yet), and if that doesn't solve the problem, it will omit the second label to improve readability.

Label 2 automatically floated to the left to avoid conflicts.

Finally, although hard to demonstrate on a screen, MapBuilder layers will now print better.  To create high resolution PDFs, CalTopo starts with a large image and then shrinks it down to fit the page.  With layers that look the same at all zoom levels, like USGS map scans, this works great.  With layers that get redrawn at each zoom level, like OpenStreetMap, shrinks all the lines and text down to the point where they're hard to read.

When you print a MapBuilder layer, the PDF generator will request a tile that's been scaled up by a custom amount so that lines weights and text will come out looking right.  This is only possible on layers produced in-house, but at some point I may ramp up an OpenStreetMap server so that I can do the same thing with OSM maps.

1:24k PDF created without scaling

1:24k PDF with scaling

Tuesday, December 1, 2015

WMS and WMTS: Using CalTopo Layers In ArcGIS

I regularly get asked about using CalTopo layers in desktop programs, often but not always ArcGIS.  There are a number of protocols for accessing seamless map data, and one of them (WMS) has been supported by CalTopo for a while.  However I always kept it closely guarded - if you asked nicely I'd provide the URL, but implore you not to share it with anyone lest a horde of ArcGIS users descend on my server.

With the introduction of paid subscriptions, I can now make this a publicly known feature, along with WMTS support for serving tilesets as-is.  As a note, this is limited to pro accounts, which seems reasonable to me as most people who need to import custom layers into a GIS program are likely paid professionals.

Access is restricted using account-specific API keys.  Click on your account name at the top of the left bar, and then the "Your Account" tab in the dialog.

If you are a pro-level user, you'll see an API access section with links to the WMS and WMTS endpoints.  In this case, ABC123 is my auto-generated (and made up) API key.

If the program you're using supports WMTS, you'll have better luck using that to serve tiles and letting your program combine them into a seamless image.  Because WMS has CalTopo's server compose a new image every time you pan the map, redraws will be slower and can't take advantage of caching.

Both services are provided in spherical mercator projection only (EPSG 3857, 900913, etc).  If you need something else, your GIS program needs to be able to do the reprojecting.

Also note that unlike traditional WMS services, CalTopo's WMS API is only intended to have one layer turned on at a time.  If you turn on multiple layers at once, you may get unexpected results, including no data showing up.

Comprehensive instructions for your program of choice should be available online, but an abbreviated version for ArcGIS and WMTS is shown below.  To start with, on the Add Data dialog, find the GIS Servers folder and choose "Add WMTS Server".

Enter the WMTS URL from your CalTopo account dialog, click Get Layers to verify it, and then OK.

Choose the layer you want to add.


As another example, here's the new MBTopo layer rendered in ArcGIS:

Sunday, November 8, 2015

MapBuilder: Pimp Your Topo

I've never been happy with CalTopo's reliance on USGS 7.5' topo scans.  While I expect them to remain the gold standard in backcountry navigation for years to come, in some places they're quite out of date.  Even when they aren't, it's difficult to use them for high level trip planning - zoom out a couple levels and it's hard to see trails or read the text.  Often I'll zoom in to trace out a trail, then zoom out to get the high level picture.

Vogelsang Pass, Yosemite.  At zoom 13, you can barely follow the trails.
I've thought about building my own map layer to address this for years.  The tools are all out there - throw an OpenStreetMap snapshot and some government datasets in a PostGIS database, write some Mapnik styles and put a tile server in front of it all.  The hard part is striking the right visual balance and making it all look good.  Should the map be crowded with details, or sparse and easy to view?  Are public/private land boundaries important, or should they be hidden?  At wider zooms, which features do you keep and which do you progressively phase out?

Much better.
On a backpacking trip this summer, I realized I was thinking about the problem wrong.  CalTopo is all about letting users customize the map they're looking at, and an in-house topo shouldn't be any different.  Instead of creating a single topo layer, I should be letting users design their own.

MapBuilder, the product of that realization, is now here - although it's still in "beta" and under active development.  Once I get things running smoothly, several pre-configured map layers will remain free, while the custom MapBuilder tool will become a pro-level feature.

As with the other custom layers, get started by choosing "Add MapBuilder Layer" under the new layer menu.  A dialog pops up with a bunch of design choices; click Save and your custom layer will show up in the dropdown next to all the others.

There are also two pre-configured layers in the Topographic Maps section of the standard layer dropdown, although their composition may be subject to change.  MB Topo looks like a basic topo map, while MB Aerial uses an aerial imagery background and drops some of the clutter, like peak names and park boundaries.

Anyway, enough with the talk, here's a quick tour.

MB Topo view of the Marin Headlands

MB Aerial rendering of the same spot

MB Topo rendering of Mt Olympus, WA
Custom Grand Canyon map that emphasizes trails.
Custom high-level planning map of Cascade Pass, WA.
Custom map of Logan Pass, Glacier National Park, MT

Some caveats:
  • The data I'm using for relief shading has some issues with wide zoom levels.  It should be fixed in a week, but until then North-facing slopes may look rough.
  • OpenStreetMap trail data includes some common cross-counry travel routes, like the Whitney MR and the DC and Emmons routes up Rainier.  So far I haven't figured out how to distinguish them from actual trails.
  • The PDF generator creates high resolution maps by zooming in and then shrinking everything down.  With layers that change at each zoom level, this makes text and features really small.  I hope to have a solution soon, so that MapBuilder layers will print acceptably.
  • Since this is under active development, existing maps may cease to work or suddenly start looking different.
  • Remember, once I get the bugs worked out, custom layers will become a pro-level feature, but several preconfigured layers will remain available.

Saturday, October 24, 2015

Subscriptions Go Live

While I don't expect this news to be greeted with thundering applause, CalTopo subscriptions are now live.  Based on feedback from y'all, I moved to a two-tier pricing model:

You can still create, share and print maps for free, but prints will be limited to 5 pages of 8.5x11 and KMZ and MBTiles exports will be size limited.

At the moment, the Pro level is simply everything the Basic level has, but bigger - maybe I should have called it "super size" instead.  Unless you really need large-format PDFs or large scale KMZ or MBTiles exports, I realize it may be a tough sell for $50 at the moment.  However there are some additional features slated for the Pro tier that I'm pretty excited about, and some of which will carry significantly increased costs, and I don't want to be stuck needing to raise prices in six months.

For the detail-oriented, there a few sneaker items in that table:

  • PDFs can now go to 48" instead of just 36".  Perfect for those with access to office plotters and ink purchased by the barrel.
  • The maximum PDF length is now 40 pages, but multipage PDFs are limited to 8.5x11 and 11x17.  If you absolutely need a 5 page 24"x36" PDF, you'll need to make five separate documents and combine them on your own.  That keeps my server much more sane.
  • The default PDF retention limit is now 7 days.  Most people don't need a linkable PDF longer than that, and PDF data was becoming a significant portion of my total DB size.
  • KMZs now go to 200 images instead of 100.  Be aware that most Garmin GPSs won't handle more than 100 images across all KMZs loaded on the device, because Garmin wants you to buy BirdsEye imagery instead.
  • MBTiles exports are now possible.  This has become the go-to tile storage format for smartphones, and I'm excited to offer it.  More on that in another post, but if you use an app that already licenses my map layers, such as Gaia or Backcountry Navigator Pro, you're generally better off downloading maps directly through them.
In addition to those user-facing features, I also upgraded (again) to a new server with twice the RAM.  For the second time in a year, CalTopo had started experiencing short, intermittent outages from running out of memory during heavy usage.

I've tried to keep nagging to a minimum and to avoid full-page "Upgrade Now!" popups.  I'm firmly convinced that CalTopo has a solid value proposition and that the site itself is my best sales tool.  Unavailable options are grayed out and have an upgrade notice nearby, but otherwise, there's plenty of opportunity for new users to discover why the site is so valuable, instead of me beating them over the head with it.

If you are signed in, and are not a subscriber, you will also see an upgrade link in the top menu:

To change your account status, or simply find out more about it, go to or click on your account in the left bar, choose the "Your Account" tab, and then the "Change Status" button.  You'll see something like this:

Payments are handled by Stripe; your credit card number goes straight to their server and I never handle or know it (other than the last 4):

Any problems, please drop me a line at  SAR members and people who have previously donated should contact me directly for a redemption code.

Monday, October 19, 2015

HTTPS Support

In preparation for subscriptions and credit card handling, CalTopo now supports secure https connections.  Because all requests from an https page must also be in https, there are a few features that don't work quite right yet, most noticeably the land management, fire activity and fire history layers, but that's all I've found so far.  If you run into other issues, please let me know.

That concludes this public service announcement.

Wednesday, September 9, 2015

CalTopo Goes Commercial

I've spent a lot of time lately thinking about the future of CalTopo.  While I'd love to claim that I started with some kind of genius original vision, the truth is that it started as an offshoot of a hobby project I'd been working on for SAR, and grew organically from there.  As with any software project, the work started out fun and gradually became more of a drag.  Now much of my time is spent answering emails, fixing bugs, and dealing with operational issues.

In short, the first 10 hours or so a week I spend on CalTopo have all the excitement of a regular job, but without the upside of getting paid.  I only get to do the fun stuff after I eat my broccoli, and many weeks that never happens.

Happier times, when CalTopo's landing page didn't have a map viewer, you had to create a map before you could draw anything, and I didn't have any support work.

After devoting my free time almost entirely to CalTopo for a number of years, I've been getting restless.  The past year saw me delve into researching the impact of terrain on lost person behavior in SAR.  While the results may seem obvious, an entire generation of search managers has been trained using the aeronautical/maritime model of gridded area searching with little regard to terrain.  The Department of Homeland Security, which fancies itself as owning inland SAR in the US, is still heavily invested in that model, as is the current crop of textbooks.  My research project is a tiny drop in the sea of effort required to restore some sanity to the way people are found, and I would like to do more.

Another consideration is my AWS hosting bill.  August cost me $2200; rounding a bit, the costs were roughly:

  • $250 to host map tiles on S3
  • $200 to host the and servers
  • $100 of random bits and pieces, such as EBS snapshots full of GIS data that I need to keep around
  • $1400 in data charges, mostly serving map tiles


There are a few smartphone apps that license the map tiles from me; they form the bulk of the $1400 in usage costs, and also pay me just enough to subsidize the entire AWS bill.  However if they were to pull out I'd be up a financial creek: beyond the fixed hosting costs, CalTopo users alone account for $100 a month worth of map data.  This is not some forum that can be run on a cheap shared host, and while I greatly appreciate the various donations people have made, they fall well short of the site's actual monthly operating costs.

The costs involved with developing and running CalTopo - both real and opportunity, financial and time wise - have been significant.  While I still cry a little inside every time I price out a 4x4 Sprinter camper van, I don't regret the hours and funds plowed into it.  However, looking to the future, how long can I continue channeling such a large part of my life towards CalTopo?

Yeah, Baby!  If I could go back in time, would I trade CalTopo for one of these bad boys in my driveway?  Decisions, decisions.

I don't know what the answer is, but it's certainly not another 5 years.  Even ignoring the pull of other projects, I've simply been spending too much time locked away behind a desk.  My own sanity requires more weekends spent recreating rather than working.

As I see it, there are basically three options:

Keep the current model and let the site stagnate.  I'll occasionally be able to make time for improvements, but the pace will slow and large features like auto-routing along trails are unlikely to get implemented.  If map licensing fees drop off, either shut down or eat the $7k/yr in unavoidable hosting costs.

Open source the code and start a nonprofit to cover costs.  I tried the open source approach with early versions of the code, and no one ever contributed anything (literally, which is why I was able to return to a closed source approach).  With the site's improved popularity it may be a different story now, but I'm not convinced that the open source approach works well with hosted sites, as opposed to software that end-users run locally, or server frameworks like Apache.

In addition to code improvements, there are also the hosting costs to worry about.  I've seen plenty of nonprofits struggle for cash; the most effective ones either have strong local appeal or some kind of national advocacy message.  A backcountry mapping / SAR nonprofit wold have neither, and it's unlikely that I would be able to do more than struggle to cover each year's hosting costs.  That said, if anyone has a pile of money and a burning desire to help the outdoor and SAR communities with it, drop me a line at

Get serious about running the site like a business.  Beyond hosting costs, try to make enough to justify continued development and investment into the site.

After a lot of thought, I've decided to go with option 3.  While there was some hesitation leading up to the decision, now that it's made, I'm pretty excited about it.  A sharper focus on commercially viable features will ultimately mean a better site for users, with me forced to spend more time on documentation and UI improvements rather than chasing shiny new ideas.  It opens up the potential to include paid map layers, such as PDFs with higher resolution aerial imagery.  And, hopefully, it will allow me to invest more hours developing new features.

The "shared maps" layer was originally driven by commercial ambitions, but now that it exists, I think it's a great feature in its own right.

Understandably, I don't expect all the users out there to share my enthusiasm.  Since I'm explaining why I need to make CalTopo commercially viable, I also want to clarify a few things:

  • CalTopo will continue to be usable as a free site; it will not be hidden wholesale behind a paywall.  As a user of the site myself, I want to be able to bring a map up on a friend's computer without having to sign in.
  • CalTopo and SARTopo will continue to be free to first responders who use them professionally, such as SAR and wildland fire.
  • Billing will be simple and transparent.  You won't have to enter a card number to start a free trial, and  won't have to time your cancelation with the end of a billing period.  You can cancel with a single click, no annoying retention process.
  • You can keep your data.  You don't have to worry about losing any of your saved maps or PDFs if you cancel your subscription.
  • If you've already made a donation, you can request to have that credited towards your account.

I'm still working out the exact details and will lay them out in another post.  However, my current thinking is a $40/year subscription that includes the following:
  • Unlimited private maps.  Free users will be limited to 5 private maps, although they can create as many publicly shared ones as they want.  If you already have more than 5 private maps, you can keep them, you just can't save any new ones until you get the total under 5.
  • Save PDF URLs for up to a year.  For free accounts, URLs will expire after 30 days.
  • Large-format PDFs.  Free accounts will be limited to 11x17.
Edit: I should have made it clearer that this is simply one option.  I'd also consider something that's cheaper but encompasses more of the site; see comment below.

One feature I'm still contemplating is a smaller cap on the number of pages a PDF can have (current limit is 15 across the board).  There may also be some additional paid PDF management functions, such as the ability to hide the QR code or point it at a custom URL.

So . . . what say ye?  What's the best price / set of features that will give me a functional free site and happy paying customers?

Friday, August 28, 2015

Up Again

At 8pm on Tuesday, I got paged out for a typically inconvenient mutual aid SAR: pack everything I need for a 72-hour deployment, sleep a couple hours, wake up at 3:45, drive 6 hours, fly into the middle of the Sierra backcountry, and hike out 40 miles.

Thanks for the lift, SEKI H-552

When I get back to civilization, my phone normally goes crazy with follow-on SAR pages.  This time, there was also no shortage of emails from worried CalTopo users.  Some people saw this:

some couldn't connect to the site at all:

some couldn't print maps, and a number sent me worried emails stating that they no longer had any maps saved to their account.

While I normally find someone to keep an eye on CalTopo for me when I leave on long personal trips, there was no time to do so before this search, and the site was down for at least a day while I was off the grid.

What Happened?

Pretty simple: the CalTopo server ran out of disk space.  This is mostly due to the tile cache - PDF generation, the view from here feature and various layers require me to fetch tiles from external sources, do some work on the server, and then send them off to the client.  Rather than fetching a map tile each time it's needed, I cache it locally on the disk.

In theory, the cache is supposed to have a maximum size and purge tiles to prevent itself from growing too large.  In reality, this isn't happening.  Since the cache gets reset every time I push a code change, this has generally not been an issue.  However with CalTopo growing, and the number of tile requests increasing proportionately, I've apparently hit the point where it will fill up in only a couple days.

Easy enough to fix, at least in the short term.

What About My Data?

The lack of space caused database connection issues, which meant that CalTopo couldn't pull back maps linked to individual accounts.  The maps were never actually gone, and I didn't have to do anything like restoring from a backup.

If something catastrophic were to happen, I back the database up to Amazon S3 nightly, so - again in theory - I should never lose more than a day's worth of changes.  I do occasionally download the nightly backups to a local drive in case my Amazon account credentials get compromised, but that would mean all kinds of problems.

If you're still concerned about data preservation, you can back up maps locally by choosing "Download CalTopo Backup" under the Export menu.  This is also a way to copy one map's contents into another, while preserving line styles, comments and so forth.

Tuesday, August 18, 2015

Please Share Your Maps

Since the beginning, CalTopo has had the ability to make maps publicly visible, i.e. someone can find the map without knowing its URL.  However all this did was add the map to an obscure page that few visited other than Google.

Today that changes with the introduction of the shared maps layer.  It's available as an overlay checkbox, along with a layer for your own maps that's visible when you sign in:

At wider zooms, each shared map is shown as a simple bounding box and center marker.  This can get a little confusing - some additional UI love may be necessary - but mousing over highlights the box and center mark.  Clicking brings up an info window with a link to that map:

Zoom in a little further and individual features get displayed:

Mouse over a feature and everything from the source map will get highlighted, giving you an idea of what that map covers.  Clicking gives you the feature's name and comments, if any, and another link back to the source map.

So how do you share your maps with other CalTopo users?  When saving a new map, just choose "publicly visible" under the "Share Your Map" section of the save dialog.  Go ahead and fill in some details in the "Tell Us About Your Map" section as well.

For maps you already have, start by clicking on your account name at the top of of the left bar.

Then go ahead and and set the sharing dropdown to "Publicly Visible" for each map you want to share.

This might be a good opportunity to visit the "Your Account" tab in the same dialog and change how your account is displayed to other users.  I'm also putting a messaging system in place so that other CalTopo users can ask questions about your maps.  If you don't want them to be able to contact you, uncheck that box.

There are only around 500 maps currently shared on CalTopo.  As that number grows I may need to change some of the implementation details to improve performance or reduce information density, but I think the basic approach is here to stay.  I'm pretty excited about it - I've already discovered a few maps that I wish I'd known about sooner.

Thursday, April 2, 2015

Improved and Printable Elevation Profiles

Elevation profiles are one of the few remaining gaps people have noticed when comparing CalTopo to desktop software.  No more.

There are now two profile modes - the existing interactive one that uses Google elevation data, and a full-page version that builds on the terrain statistics dialog with CalTopo sourced elevation data.  You can get to the new profiles using the terrain statistics menu option, or clicking the expand link at the top right of the interactive profile dialog:

The expanded version is a full-page dialog:

One of the most obvious differences is axes in place of rise/run data:

There's also a heatmap below the profile showing slope angle, land cover and tree cover along the route:

Below that is the previously existing terrain stats content (lines only - terrain stats along a polygon's perimeter, to match the profile, would be meaningless or misleading).  A more subtle addition is that nearby markers are automatically placed along the profile:

Using bulk ops, you can also plot multiple lines on the same axes:

Absolute plots the lines' actual elevations, while relative shows how each line changes from its starting elevation.  Below are 3 routes plotted against each other in absolute mode (top) and relative (bottom).

PDF versions of any profile are available using the print link at the top right of the dialog.

Note that because the interactive profile uses Google data and the expanded profile uses CalTopo data, gross gain and loss numbers may be slightly different.  Like terrain stats, expanded elevation profiles will only work within the continental US.