Thursday, May 4, 2017

Introducing the GPSIO Extension

Most newer GPSs support USB Mass Storage, which means that when plugged into your computer, they show up as a drive with all your routes, tracks and waypoints saved as GPX files.  Older models, such as the Garmin 60csx still widely used in SAR, require a special protocol to transfer your data.

For websites like CalTopo, the only option for getting data off older Garmin GPSs, and the most convenient option for newer ones, was the Garmin Communicator Plugin (GCP).  Browsers have slowly been dropping support for the NPAPI plugin architecture on which GCP was based, but the final nail in its coffin was Garmin not only discontinuing support, but deciding to no longer issue new API keys.  Since I only had an API key for, when a Google Chrome security change forced me to move most traffic over to https several months ago, GCP became largely useless.

That was motivation enough to wrap up a long-simmering project: the GPSIO browser extension.  Compatible with both Chrome and Firefox, GPSIO acts as a conduit between your browser and the open-source GPSBabel program, which does the actual hard work of talking to your GPS.

GPSIO is still a little rough around the edges, but you are cordially invited to take it for a spin, report any problems, or - since it's open source - fix them yourself.  Although GPSIO only supports Garmin GPSs at the moment, I would welcome code contributions to expand it to work with the wide range of GPS types that GPSBabel supports.

For more information, see

Once installed, using GPSIO with CalTopo is pretty simple: under the Import or Export menu, click the "Connect via GPSIO" link.

Sunday, April 16, 2017

MapBuilder: It's Back And Better Than Ever

CalTopo has been having some performance issues lately, no point in trying to pretend otherwise.

Behind the scenes, there two different platforms: a Python/Mapnik/PostGIS stack that powers MapBuilder and auto-routing, and a Java app for everything else.  MapBuilder has been struggling for a while, but lately that started bleeding over into the rest of the site's operations - as the number of pending MapBuilder requests grew and grew, the Java side of the house (which proxies the requests) would eventually get overwhelmed and pages would stop loading.  A week ago I rolled out a new server and a new copy of the MapBuilder database with fresh OpenStreetMap updates, and everything pretty much came screeching to a halt.

This left me in a bit of a pickle.  If I rolled the deployment back, things would temporarily improve, but my production environment would be out of sync with my new development laptop, which would make it harder to debug the issues and arrive at a permanent fix.  If I didn't roll the change back, fixing the problem would be a lot easier, but it would be a rougher ride until then.  I bumped the primary EC2 instance up to a $100/day m4.16xlarge to buy some slack, rolled up my sleeves, and got to work.

What followed was a blurry week of long hours, coffee, beer, testing in production, and marathon coding sessions that produced results brilliantly functional yet barely understandable when re-examined the next day.  Kind of like being a CS undergrad during end-of-semester crunch time.

At any rate, the MapBuilder rendering stack has been almost completely rewritten, sorry for any inconveniences over the past several weeks, and I'm confident at this point that the latest round of performance issues are solidly in the rear view mirror.  At least, confident enough to roll out two new features that were waiting on this.

First, as a subscriber-only feature (all levels), CalTopo now offers Retina/HiDPI tiles for most layers.  Rather than rendering out a separate 512px tileset, my server pulls 4 tiles from the next zoom level down and rolls them up into a single tile.  As soon as you sign in to your account, the map viewer will automatically pull 512px tiles instead of the standard 256px ones.  If this is problematic for some reason (requests get routed though my server rather than direct to S3, so there are potential speed implications), it can be disabled via the config menu - look for the "Retina/HiDPI" option at the bottom of the Display group.

The other big news is the addition of a "hybrid shading" option to MapBuilder, and a new prefab layer based on it named MapBuilder Hybrid.  Hybrid shading blends the green-gradient canopy shading used by MapBuilder Topo with aerial imagery, producing a backdrop that shows you much of the detail you'd get from imagery, but with less of the visual noise that makes it hard to pick out other map features.

MapBuilder Hybrid, Tuolumne Meadows.
Zooming in, you can make out individual trees, but they don't overwhelm the image or distract from the trails and contours.

Example from the Casacades

Produces a nice visual effect at wide zoom levels, too.

Admittedly I'm both biased and a little loopy from the past week, but I have the MapBuilder Hybrid layer loaded on a 4k monitor and I can't stop staring at it.  I've been wanting to add this for a while and am psyched to get it live, although this is an early version that will probably see stylistic tweaks over time.

Tuesday, March 7, 2017

CalTopo Offline

In the very beginning, I started what would ultimately become CalTopo as a hobby project focused on offline environments - a command post or tailgate somewhere off in the woods far away from the nearest internet connection.  As CalTopo grew, that ability for entirely self-contained operation has remained as a little known but fundamental building block of the codebase.  Flip out a few of the underlying bits - Google Maps with OpenLayers, a standalone database with HSQLDB, Tomcat with Winstone - and you'd get a free-standing Java web app with no external dependencies.

For search and rescue, those who knew the secret handshake could get a similarly configured copy of SARTopo, delivered via physical thumb drives along with statewide map data.  While this system worked for a while, it's been chugging along on a donut spare and one headlight for some time.  Although releasing copies of a subscription-driven website into the wild feels a bit risky, it's past time to make the offline functionality a supported feature that doesn't require my involvement with every install.

Welcome to CalTopo (and SARTopo) Offline.

Due to the costs of delivering map data and the extra support footprint it will present, there was no way I could roll the offline functionality into the existing subscription levels.  Instead there is now a $100/yr offline subscription level that also includes all pro-level features.  First responders who have upgraded to a SAR account will see a $50/yr offline upgrade option.

After upgrading, your account dialog will have an offline access tab with links to downloading both map data and a copy of the program:

Map data is provided in 1 degree by 1 degree blocks, in MBTiles format.  Available layers include USGS 7.5' topos, forest service maps, slope angle shading, elevation data for profiles and viewsheds, and NAIP aerial imagery down to 1 meter resolution; most blocks include 2 years' worth of imagery.

Each account is allowed 300GB worth of map downloads per year, with no geographic restrictions other than being limited to the lower 48 states.   It's up to you whether you want to skip the imagery and cover a larger footprint, or get 2 years worth of 1 meter imagery for a smaller coverage area.  At the moment data is only available for the red-shaded areas below, but that should grow to full lower-48 coverage within another week or two.

I'd post some screenshots, but there's not much to show - the offline version looks very much like  For more information on how this all works, check out some of the very thorough documentation written Patty Lindsay (there is hopefully more documentation of this caliber coming to

Saturday, January 7, 2017

New Year, New Layers

In my last blog post, I mentioned that I spent the tail end of 2016 cooking up some new map layers.  Well, here we go.

Terrain Shading and Custom Relief

In SAR, it's not uncommon to have a group of people huddled around a large-format map, looking at it from all angles.  While relief shading helps terrain features pop, it only works when the map is viewed from the bottom - standing at the top will cause features to invert with peaks looking like valleys.

One alternative is terrain shading (for lack of a better word), where the relief is generated using a number of different light sources, rather a single 315 degree (NW) angle.  Because of the multiple angles used, there is no "up" or "down", and the map can be viewed from any angle without playing tricks on the eye.  The downside is that in some areas it can be hard for the eye to quickly tell up from down.

For CalTopo's terrain shading, I tried to balance these tradeoffs by using 6 evenly spaced light sources plus an additional one at 315 degrees, proving a slight amount of orientation.

CalTopo enhanced relief, which uses a multiply blend filter but retains the standard
315 degree light source

CalTopo terrain relief, which uses a 7 different light sources
If the out-of-the-box terrain shading doesn't quite work for you, there's now an "Add Custom Relief" option under the Add New Layer menu.  Pick an azimuth (compass direction) and zenith (angle above the horizon) and click "Add Lighting".  Mix and match multiple combinations to reach your desired effect.

FSTopo 2016

The FSTopo maps behind the "US Forest Service" map layer have been seeing regular updates, but it's not all forward progress.  While the newer maps gained vegetation shading and better road/trail data, the delineation between public and private property moved from light, transparent gray shading to a heavy-handed gray that completely obscured all vegetation shading.  I wasn't happy with the way this looked, and wasn't happy dropping public/private land boundaries, so I did things the CalTopo way: create a new layer.  And thus the USFS 2016 layer was born.

For most purposes the now-renamed "FSTopo (2013)" and "FSTopo (2016)" layers will be interchangeable.  The old layer is better for locating land boundaries, and its white background works well for blending with aerial imagery or slope angle shading.  With its vegetation shading, the new layer is probably better suited for standalone use.

The same map as above, but this time using the "FSTopo (2016)" layer.

Expanded Alaska DEM Coverage

While not a new layer per se, I've been slowly expanding CalTopo's Alaska coverage.  Although the national elevation dataset (NED) still only covers a portion of the state, that portion has been growing, and it's time to catch up.  The NED isn't a first-class layer, but a powers a lot of other ones, including normal relief, enhanced relief, 40' contours, fixed and gradient slope angle shading, custom DEM shading, and cursor point elevations.

The current state of NED-based layer coverage in Alaska.
Note: I'm still patching up a few small errors in the dataset, but didn't want to hold this announcement up until those were fixed.

New NAIP Imagery Layer

The new layers based on the National Agriculture Imagery Program (NAIP) dataset are probably worthy of a blog post all their own.  But before I get into the good stuff, know that all this layer creation doesn't come cheap, and high quality imagery is definitely the worst offender.  Not that I'm complaining or looking for your sympathy, but lest there be any doubt as to whether your subscription dollars are getting rolled back into CalTopo development:

By way of background, while Google's Satellite layer is great, I can't do any server-side manipulation on it, whether that's generating PDFs or combining it with enhanced relief shading.  Although a secondary issue, I also can't provide offline copies of it for use by SAR teams in remote environments.

NAIP data has always been public, but in the past, acquiring it was a bit awkward, requiring the mailing of many terabytes of drives back and forth.  I tried to skirt around this by stitching the aerial backgrounds from the USGS's new "USTopo" maps into a seamless layer called USTopo Imagery.  This worked, but the quality wasn't great, and update cycles were delayed vs going directly to the source.  I've been on the hunt for some time, and as drive costs dropped, was seriously considering biting the bullet and acquiring a physical copy.

Then Andrew Johnson from Gaia GPS pointed me at the aws-naip public S3 bucket, and it was off the races.  Yes, this was expensive.  Yes, it's a roll of the dice when big-name companies like Google, MapBox and ESRI have better datasets out there.  From a business perspective, maybe it won't work out.  But I believe that high quality aerial imagery that I can repurpose as needed is strategically vital to CalTopo's future, so I decided to roll the dice and here we are.

The biggest difference between the NAIP and USTopo Imagery layers is quality.  The NAIP layer goes to zoom 17 (~1m per pixel) while USTopo Imagery went to 16 (~2m per pixel), but even at zoom 16 there's a noticeable quality difference between the two.

USTopo Imagery view of "Half Dome Village" aka Curry Village, Yosemite NP
Same location viewed using the NAIP layer

NAIP is generated in 3 year cycles, i.e. one third of the continental US is overflown each year.  Not content with a single NAIP layer, I generated two versions - one for 2011 to 2013 and one for 2013-2015.  Most places in the continental US should have two different dates available, either so that you can see how things have changed with time, or in case one revision has too much snow, shadows in the wrong place, etc.  Long term, I hope to grow the date range.

Prior imagery of the same location.  In this case, it's not much different.

NAIP is also distributed as 4-band imagery, with a near infrared channel in addition to the standard red, green and blue.  I captured this and rendered it out into a separate layer, which allows for some interesting data processing.  Right now, I'm still conflicted as to whether it's actually useful or just a neat party trick.

The Aerial Imagery section of the layer dropdown now has a "False Color IR" option.  This uses the near IR channel for red, red for blue, blue for green, and drops the green entirely.  As a result, the difference between near IR and IR is accentuated, drawing stark contrast between vegetation and manmade, dirt or rock surfaces, regardless of actual color.

False-color IR view.  No, it's not some weird 3D glasses thing.

The computed difference between red and near IR is also available as a vegetation shading option for custom MapBuilder layers, called "Infrared Reflectance".  With this option you can generate traditional looking topo maps with super-accurate vegetation shading, but as always there's a catch: areas that were shadowed in the original image show as white rather than the appropriate vegetation shading.

Custom MapBuilder layer with the IR Reflectance background.  Note the white band in the meadow at the top of the picture, which is shaded in the original image, not actually vegetation-free.
Deprecation of Existing Layers

With the layer dropdown getting increasingly complicated, this was also a good time to clean house.  The "ArcGIS Topo" option isn't as clean as my USGS map scans, but I originally included it because it covered Alaska.  That's no longer an issue, so it's gone.  USGS 1:250k maps aren of limited utility with Google Terrain and MapBuilder Topo; gone.  USTopo Imagery is inferior to the NAIP layer in pretty much every way; gone.  CA Visitor Maps had some visitor maps that can't be found elsewhere, but I need to move past state-specific layers in the dropdown, and I hope to grow the NPS and USFS visitor maps soon to help make up the gap.

All of these layers are still accessible in two ways.  First, any existing maps or links that referenced them will continue to work, although they won't display properly in the layer dropdown.  Second, they're all available as prefill custom sources.  Click on Add New Layer -> Add Custom Source, and choose the layer you want out of the "prefill with" dropdown.

Use a particular layer a lot?  Save it to your account so that it's always available.

Monday, January 2, 2017

2016 Q3/Q4 In Review

The CalTopo blog has been quiet since spring, but that doesn't mean a lack of progress, much less a lack of work.  Time to take a quick look back at the second half of 2016.

First, the personal front.  CalTopo has been my full-time job since May, and although it's averaged more than 40 hours per week, I did manage to mix a bit of vacation in.  Some of it traditionally enjoyable:

W Ridge of Pigeon Spire, Bugaboos

And some of it just grueling:

Leading the morning briefing on a campaign search after a full week of 18-20hr days.

However the universal theme for the summer, and the reason for the lack of blog updates, was that I was simply trying to stay afloat.  In between email deluges, that meant tracking down some scaling and performance issues that would always seem to trigger a crash and site outage at the most inconvenient times.

By fall, I had the performance issues sorted out, and decreasing seasonal usage lessened my customer support workload.  So it was time to get cranking on improvements.

Two lines simultaneously open for editing.
The largest of those was a major UI overhaul, moving most editing from modal, bottom-of-screen dialogs to modeless ones that stack up on the side of the screen.  This brought with it a number of improvements, including massive performance increases for large datasets, per-object visibility toggling, simultaneous editing of multiple objects (such as neighboring polygons), instant syncing of line/marker style to the map, and single-step drawing (the old UI required you to choose a style, hit OK, and then start drawing).

Per-object visibility makes it easier to clean up a cluttered map.

The other changes were much smaller but still much-needed.  Auto-drawing now has the option for larger lines (I need to make this the default), which helps prevent the oft-recurring problem of accidentally clicking next to the stream or trail rather than on it.

Auto-drawing with larger line widths.

DEM (digital elevation model) shading allowed the creation of custom shaded layers based on slope angle, elevation and aspect, but required understanding a cryptic syntax (such as s30-60e4000-6000f FF0000).  This has been replaced with a friendlier dialog that allows you to select values from dropdowns and colors from a color picker:

Preferences, such as a user's preferred datum and coordinate system, used to be stored as a browser cookie.  By storing just an ID cookie and tracking preferences in a server-side database, I can store more information without risking exceeding the cookie size limit.  This allowed me to expand to the print page, remembering a user's last selected page size, scale, and other features.

WMS and WMTS layers have been supported for a while, but reverse engineering the WMS request syntax was tricky for casual users.  The Add Custom Layer dialog now has an auto-probe option that will talk to a WMS or WMTS endpoint and try to configure the URL template automatically, making it easier to pull more third-party sources into CalTopo, particularly government run servers with a wealth of public domain (but limited geographic coverage) data.

As well as lots of smaller changes not worth listing.  That brings us to about November, when I switched gears and began working on map data instead of features.  Those are for a subsequent post, but suffice to say some exciting changes have recently gone live.

Monday, May 30, 2016

Downloadable Garmin GPS Maps

At the beginning of 2015, I tried selling statewide BirdsEye(TM)-format maps on MicroSD cards.  Because Garmin locks your GPS's firmware, you need to install an unlocked copy in order get maps from anywhere other than their paid BirdsEye subscription service.  A naked cash grab if there ever was one; Garmin would apparently prefer to milk its captive market while it can, rather than take steps which might delay or prevent smartphones from eating the consumer GPS market for lunch.

At any rate, the locked firmware is enough to prevent someone like me from selling map cards through a major retailer.  Sales from the CalTopo store were never huge, and always seemed to come at inconvenient times, like when I was on vacation or similarly indisposed.  Because I was selling physical products, an annoying degree of paperwork was required to report marginal amounts of sales tax.  I was always worried about losing a thumbdrive in the mail, or sending it to the wrong address due to a typo.

It's past time for me to move away from selling physical products.  Instead, all of the JNX files I'd sold on thumbdrives and SD cards are now available as downloads to pro-level subscribers.  Simply click on the new downloads tab of your account dialog, and then on an individual layer link

Newer maps are broken up into 1 degree by 1 degree blocks, and named based on the southeast corner.  Older maps have custom boundaries as illustrated here and here.

Although this is only available to pro-level subscribers, a 1-year pro subscription costs the same as the SD cards I was selling, but gives you access to all the maps rather than just one state.


Friday, May 27, 2016

Go North - Alaska Gets (Partial) Elevation Data

The USGS has been adding portions of Alaska to the National Elevation Dataset (NED), and although coverage isn't anywhere near complete, I finally decided it was time to pull the trigger on an initial buildout of elevation data for Alaska.

current coverage footprint
The buildout includes:

  • Normal and enhanced relief
  • 40' contours
  • Elevation data
  • DEM shading, viewsheds and sunlight analysis
  • View from here
  • Fixed and gradient slope shading

Denali looks kind of mellow until you turn on slope angle shading, at which point the West Buttress route becomes a little more obvious.

My viewshed and sunlight layers, as well as the view from here feature, assume that nothing is higher than about 14k, as a performance optimization feature.  So for the moment, the view of Denali from Hunter looks a little messed up:

I still don't have canopy and or land coverage data for Alaska or Hawaii, so while I can display terrain stats, those fields are blank: