Saturday, January 7, 2017

New Year, New Layers

In my last blog post, I mentioned that I spent the tail end of 2016 cooking up some new map layers.  Well, here we go.

Terrain Shading and Custom Relief

In SAR, it's not uncommon to have a group of people huddled around a large-format map, looking at it from all angles.  While relief shading helps terrain features pop, it only works when the map is viewed from the bottom - standing at the top will cause features to invert with peaks looking like valleys.

One alternative is terrain shading (for lack of a better word), where the relief is generated using a number of different light sources, rather a single 315 degree (NW) angle.  Because of the multiple angles used, there is no "up" or "down", and the map can be viewed from any angle without playing tricks on the eye.  The downside is that in some areas it can be hard for the eye to quickly tell up from down.


For CalTopo's terrain shading, I tried to balance these tradeoffs by using 6 evenly spaced light sources plus an additional one at 315 degrees, proving a slight amount of orientation.

CalTopo enhanced relief, which uses a multiply blend filter but retains the standard
315 degree light source


CalTopo terrain relief, which uses a 7 different light sources
If the out-of-the-box terrain shading doesn't quite work for you, there's now an "Add Custom Relief" option under the Add New Layer menu.  Pick an azimuth (compass direction) and zenith (angle above the horizon) and click "Add Lighting".  Mix and match multiple combinations to reach your desired effect.



FSTopo 2016

The FSTopo maps behind the "US Forest Service" map layer have been seeing regular updates, but it's not all forward progress.  While the newer maps gained vegetation shading and better road/trail data, the delineation between public and private property moved from light, transparent gray shading to a heavy-handed gray that completely obscured all vegetation shading.  I wasn't happy with the way this looked, and wasn't happy dropping public/private land boundaries, so I did things the CalTopo way: create a new layer.  And thus the USFS 2016 layer was born.

For most purposes the now-renamed "FSTopo (2013)" and "FSTopo (2016)" layers will be interchangeable.  The old layer is better for locating land boundaries, and its white background works well for blending with aerial imagery or slope angle shading.  With its vegetation shading, the new layer is probably better suited for standalone use.

The same map as above, but this time using the "FSTopo (2016)" layer.

Expanded Alaska DEM Coverage

While not a new layer per se, I've been slowly expanding CalTopo's Alaska coverage.  Although the national elevation dataset (NED) still only covers a portion of the state, that portion has been growing, and it's time to catch up.  The NED isn't a first-class layer, but a powers a lot of other ones, including normal relief, enhanced relief, 40' contours, fixed and gradient slope angle shading, custom DEM shading, and cursor point elevations.

The current state of NED-based layer coverage in Alaska.
Note: I'm still patching up a few small errors in the dataset, but didn't want to hold this announcement up until those were fixed.

New NAIP Imagery Layer

The new layers based on the National Agriculture Imagery Program (NAIP) dataset are probably worthy of a blog post all their own.  But before I get into the good stuff, know that all this layer creation doesn't come cheap, and high quality imagery is definitely the worst offender.  Not that I'm complaining or looking for your sympathy, but lest there be any doubt as to whether your subscription dollars are getting rolled back into CalTopo development:


By way of background, while Google's Satellite layer is great, I can't do any server-side manipulation on it, whether that's generating PDFs or combining it with enhanced relief shading.  Although a secondary issue, I also can't provide offline copies of it for use by SAR teams in remote environments.

NAIP data has always been public, but in the past, acquiring it was a bit awkward, requiring the mailing of many terabytes of drives back and forth.  I tried to skirt around this by stitching the aerial backgrounds from the USGS's new "USTopo" maps into a seamless layer called USTopo Imagery.  This worked, but the quality wasn't great, and update cycles were delayed vs going directly to the source.  I've been on the hunt for some time, and as drive costs dropped, was seriously considering biting the bullet and acquiring a physical copy.

Then Andrew Johnson from Gaia GPS pointed me at the aws-naip public S3 bucket, and it was off the races.  Yes, this was expensive.  Yes, it's a roll of the dice when big-name companies like Google, MapBox and ESRI have better datasets out there.  From a business perspective, maybe it won't work out.  But I believe that high quality aerial imagery that I can repurpose as needed is strategically vital to CalTopo's future, so I decided to roll the dice and here we are.

The biggest difference between the NAIP and USTopo Imagery layers is quality.  The NAIP layer goes to zoom 17 (~1m per pixel) while USTopo Imagery went to 16 (~2m per pixel), but even at zoom 16 there's a noticeable quality difference between the two.

USTopo Imagery view of "Half Dome Village" aka Curry Village, Yosemite NP
Same location viewed using the NAIP layer

NAIP is generated in 3 year cycles, i.e. one third of the continental US is overflown each year.  Not content with a single NAIP layer, I generated two versions - one for 2011 to 2013 and one for 2013-2015.  Most places in the continental US should have two different dates available, either so that you can see how things have changed with time, or in case one revision has too much snow, shadows in the wrong place, etc.  Long term, I hope to grow the date range.

Prior imagery of the same location.  In this case, it's not much different.

NAIP is also distributed as 4-band imagery, with a near infrared channel in addition to the standard red, green and blue.  I captured this and rendered it out into a separate layer, which allows for some interesting data processing.  Right now, I'm still conflicted as to whether it's actually useful or just a neat party trick.

The Aerial Imagery section of the layer dropdown now has a "False Color IR" option.  This uses the near IR channel for red, red for blue, blue for green, and drops the green entirely.  As a result, the difference between near IR and IR is accentuated, drawing stark contrast between vegetation and manmade, dirt or rock surfaces, regardless of actual color.

False-color IR view.  No, it's not some weird 3D glasses thing.

The computed difference between red and near IR is also available as a vegetation shading option for custom MapBuilder layers, called "Infrared Reflectance".  With this option you can generate traditional looking topo maps with super-accurate vegetation shading, but as always there's a catch: areas that were shadowed in the original image show as white rather than the appropriate vegetation shading.

Custom MapBuilder layer with the IR Reflectance background.  Note the white band in the meadow at the top of the picture, which is shaded in the original image, not actually vegetation-free.
Deprecation of Existing Layers

With the layer dropdown getting increasingly complicated, this was also a good time to clean house.  The "ArcGIS Topo" option isn't as clean as my USGS map scans, but I originally included it because it covered Alaska.  That's no longer an issue, so it's gone.  USGS 1:250k maps aren of limited utility with Google Terrain and MapBuilder Topo; gone.  USTopo Imagery is inferior to the NAIP layer in pretty much every way; gone.  CA Visitor Maps had some visitor maps that can't be found elsewhere, but I need to move past state-specific layers in the dropdown, and I hope to grow the NPS and USFS visitor maps soon to help make up the gap.

All of these layers are still accessible in two ways.  First, any existing maps or links that referenced them will continue to work, although they won't display properly in the layer dropdown.  Second, they're all available as prefill custom sources.  Click on Add New Layer -> Add Custom Source, and choose the layer you want out of the "prefill with" dropdown.

Use a particular layer a lot?  Save it to your account so that it's always available.

Monday, January 2, 2017

2016 Q3/Q4 In Review

The CalTopo blog has been quiet since spring, but that doesn't mean a lack of progress, much less a lack of work.  Time to take a quick look back at the second half of 2016.

First, the personal front.  CalTopo has been my full-time job since May, and although it's averaged more than 40 hours per week, I did manage to mix a bit of vacation in.  Some of it traditionally enjoyable:

W Ridge of Pigeon Spire, Bugaboos

And some of it just grueling:

Leading the morning briefing on a campaign search after a full week of 18-20hr days.

However the universal theme for the summer, and the reason for the lack of blog updates, was that I was simply trying to stay afloat.  In between email deluges, that meant tracking down some scaling and performance issues that would always seem to trigger a crash and site outage at the most inconvenient times.

By fall, I had the performance issues sorted out, and decreasing seasonal usage lessened my customer support workload.  So it was time to get cranking on improvements.

Two lines simultaneously open for editing.
The largest of those was a major UI overhaul, moving most editing from modal, bottom-of-screen dialogs to modeless ones that stack up on the side of the screen.  This brought with it a number of improvements, including massive performance increases for large datasets, per-object visibility toggling, simultaneous editing of multiple objects (such as neighboring polygons), instant syncing of line/marker style to the map, and single-step drawing (the old UI required you to choose a style, hit OK, and then start drawing).

Per-object visibility makes it easier to clean up a cluttered map.

The other changes were much smaller but still much-needed.  Auto-drawing now has the option for larger lines (I need to make this the default), which helps prevent the oft-recurring problem of accidentally clicking next to the stream or trail rather than on it.

Auto-drawing with larger line widths.

DEM (digital elevation model) shading allowed the creation of custom shaded layers based on slope angle, elevation and aspect, but required understanding a cryptic syntax (such as s30-60e4000-6000f FF0000).  This has been replaced with a friendlier dialog that allows you to select values from dropdowns and colors from a color picker:


Preferences, such as a user's preferred datum and coordinate system, used to be stored as a browser cookie.  By storing just an ID cookie and tracking preferences in a server-side database, I can store more information without risking exceeding the cookie size limit.  This allowed me to expand to the print page, remembering a user's last selected page size, scale, and other features.

WMS and WMTS layers have been supported for a while, but reverse engineering the WMS request syntax was tricky for casual users.  The Add Custom Layer dialog now has an auto-probe option that will talk to a WMS or WMTS endpoint and try to configure the URL template automatically, making it easier to pull more third-party sources into CalTopo, particularly government run servers with a wealth of public domain (but limited geographic coverage) data.

As well as lots of smaller changes not worth listing.  That brings us to about November, when I switched gears and began working on map data instead of features.  Those are for a subsequent post, but suffice to say some exciting changes have recently gone live.

Monday, May 30, 2016

Downloadable Garmin GPS Maps

At the beginning of 2015, I tried selling statewide BirdsEye(TM)-format maps on MicroSD cards.  Because Garmin locks your GPS's firmware, you need to install an unlocked copy in order get maps from anywhere other than their paid BirdsEye subscription service.  A naked cash grab if there ever was one; Garmin would apparently prefer to milk its captive market while it can, rather than take steps which might delay or prevent smartphones from eating the consumer GPS market for lunch.

At any rate, the locked firmware is enough to prevent someone like me from selling map cards through a major retailer.  Sales from the CalTopo store were never huge, and always seemed to come at inconvenient times, like when I was on vacation or similarly indisposed.  Because I was selling physical products, an annoying degree of paperwork was required to report marginal amounts of sales tax.  I was always worried about losing a thumbdrive in the mail, or sending it to the wrong address due to a typo.

It's past time for me to move away from selling physical products.  Instead, all of the JNX files I'd sold on thumbdrives and SD cards are now available as downloads to pro-level subscribers.  Simply click on the new downloads tab of your account dialog, and then on an individual layer link


Newer maps are broken up into 1 degree by 1 degree blocks, and named based on the southeast corner.  Older maps have custom boundaries as illustrated here and here.

Although this is only available to pro-level subscribers, a 1-year pro subscription costs the same as the SD cards I was selling, but gives you access to all the maps rather than just one state.

 


Friday, May 27, 2016

Go North - Alaska Gets (Partial) Elevation Data

The USGS has been adding portions of Alaska to the National Elevation Dataset (NED), and although coverage isn't anywhere near complete, I finally decided it was time to pull the trigger on an initial buildout of elevation data for Alaska.

current coverage footprint
The buildout includes:

  • Normal and enhanced relief
  • 40' contours
  • Elevation data
  • DEM shading, viewsheds and sunlight analysis
  • View from here
  • Fixed and gradient slope shading


Denali looks kind of mellow until you turn on slope angle shading, at which point the West Buttress route becomes a little more obvious.

My viewshed and sunlight layers, as well as the view from here feature, assume that nothing is higher than about 14k, as a performance optimization feature.  So for the moment, the view of Denali from Hunter looks a little messed up:



I still don't have canopy and or land coverage data for Alaska or Hawaii, so while I can display terrain stats, those fields are blank:


Wednesday, May 4, 2016

New Help System

Answering emails has become a growing part of my daily routine, and I've been looking for ways to cut back.  A support forum felt like the obvious answer, but I didn't want to force people to signup for forum accounts, and it would be nice if private account issues could be handled in the same place as public UI questions.

I recently discovered Tender (tenderapp.com, not to be confused with the food-dating tender smartphone app), and it looks like a good fit.  Forum-based discussions that anyone can chime in on, email integration, and the ability to set conversations as either public or private.


There's now a new help and support site up at help.caltopo.com, with a discussion forum and soon-to-be-expanded knowledge base.  Once the kinks are worked out, I'll start routing emails there as well, so that I can make useful conversations public.

Friday, April 1, 2016

New Zealand Topo Coverage

One of the advantages of shifting to a paid, commercial model is that it gives you cash to spend on things.  In one my first expenditures beyond CalTopo's regular operating costs, I've acquired a copy of nztopomaps.com's tile set, expanding scanned topo coverage to New Zealand.


As with coverage for Canada, none of the elevation based layers - shaded relief, slope angle shading, etc - work in New Zealand.  You're limited to the government produced topo layer, although the small / interactive profile will still work as it uses Google's elevation data.

As with Canada, these maps have been seamlessly integrated with the "USGS 7.5' Topos" layer; there isn't a separate NZ-specific option.  No comment as to when or if CalTopo might expand to other countries, although it certainly helps if there are ready-to-go tilesets, rather than me having to stitch together innumerable PDF scans.

Wednesday, March 9, 2016

Auto Routing

While the freehand drawing tool helps, it's never been easy to plan long-distance trail hikes using CalTopo.  The only two ways to get realistic mileage estimates were to painstakingly trace over the map, or import someone else's GPS tracks.

By piggybacking on the MapBuilder database, CalTopo now has an auto-routing feature.  There are still a few quirks to work out, and at least until that happens, I'm leaving the feature off by default.

Start by creating a line, the same way as always:



Once you're in drawing mode, there's a new "snap to" option at the top right of the screen, next to the layer control:



As the names suggest, None turns the feature off, OSM uses OpenStreetMap data, and USFS uses Forest Service data.  With auto-routing enabled, roads and trails show on the map as semi-transparent black lines:


To begin routing, click one of the segments (it will turn red when you mouse over):



Once you are snapped to a segment, mousing over additional segments will automatically create a routable line.



Mouse back out and you'll see the traditional straight drawing tool:


Unlike typical routing tools that require you to specify start, end and mid points, this makes it easy to jump between trails and cross-country travel within the same route.  Some of the known quirks at this point include:

  • Roads and trails need to have a common vertex for the auto-router to recognize that they intersect, so some of the solutions it comes up with will not be optimal travel routes.  Think of it as "auto drawing" rather than "auto routefinding".
  • Some OSM trailheads have small gaps between the road and the trail; the auto-router will not properly jump between these.
  • For performance reasons, only 2000 lines are pulled back at once.  At wide zoom levels, the road/trail network shown will probably be incomplete.
  • Computations can lock up the browser for several seconds, especially when scrolling to a new coverage area.   Clicking on the map during these lock-ups causes the auto-router to loose track of the line you were snapped to.

Accurate trail mileages are an issue that comes up constantly, and there's no easy answer.  However, I thought the JMT would be a good test case.  In the summer of 2014, I had to plot a JMT route by modifying some PCT GPS tracks available online.  This time I was able to trace it out in a couple minutes using the auto-routing tool; the slowest part was making sure I stayed on the JMT proper rather than a nearby variation.

routing mid-trail

The CalTopo-drawn line was 210 miles to Whitney Portal, vs the official number of 221.  So about 5% short, which doesn't seen too bad.