Saturday, January 7, 2017

New Year, New Layers

In my last blog post, I mentioned that I spent the tail end of 2016 cooking up some new map layers.  Well, here we go.

Terrain Shading and Custom Relief

In SAR, it's not uncommon to have a group of people huddled around a large-format map, looking at it from all angles.  While relief shading helps terrain features pop, it only works when the map is viewed from the bottom - standing at the top will cause features to invert with peaks looking like valleys.

One alternative is terrain shading (for lack of a better word), where the relief is generated using a number of different light sources, rather a single 315 degree (NW) angle.  Because of the multiple angles used, there is no "up" or "down", and the map can be viewed from any angle without playing tricks on the eye.  The downside is that in some areas it can be hard for the eye to quickly tell up from down.

For CalTopo's terrain shading, I tried to balance these tradeoffs by using 6 evenly spaced light sources plus an additional one at 315 degrees, proving a slight amount of orientation.

CalTopo enhanced relief, which uses a multiply blend filter but retains the standard
315 degree light source

CalTopo terrain relief, which uses a 7 different light sources
If the out-of-the-box terrain shading doesn't quite work for you, there's now an "Add Custom Relief" option under the Add New Layer menu.  Pick an azimuth (compass direction) and zenith (angle above the horizon) and click "Add Lighting".  Mix and match multiple combinations to reach your desired effect.

FSTopo 2016

The FSTopo maps behind the "US Forest Service" map layer have been seeing regular updates, but it's not all forward progress.  While the newer maps gained vegetation shading and better road/trail data, the delineation between public and private property moved from light, transparent gray shading to a heavy-handed gray that completely obscured all vegetation shading.  I wasn't happy with the way this looked, and wasn't happy dropping public/private land boundaries, so I did things the CalTopo way: create a new layer.  And thus the USFS 2016 layer was born.

For most purposes the now-renamed "FSTopo (2013)" and "FSTopo (2016)" layers will be interchangeable.  The old layer is better for locating land boundaries, and its white background works well for blending with aerial imagery or slope angle shading.  With its vegetation shading, the new layer is probably better suited for standalone use.

The same map as above, but this time using the "FSTopo (2016)" layer.

Expanded Alaska DEM Coverage

While not a new layer per se, I've been slowly expanding CalTopo's Alaska coverage.  Although the national elevation dataset (NED) still only covers a portion of the state, that portion has been growing, and it's time to catch up.  The NED isn't a first-class layer, but a powers a lot of other ones, including normal relief, enhanced relief, 40' contours, fixed and gradient slope angle shading, custom DEM shading, and cursor point elevations.

The current state of NED-based layer coverage in Alaska.
Note: I'm still patching up a few small errors in the dataset, but didn't want to hold this announcement up until those were fixed.

New NAIP Imagery Layer

The new layers based on the National Agriculture Imagery Program (NAIP) dataset are probably worthy of a blog post all their own.  But before I get into the good stuff, know that all this layer creation doesn't come cheap, and high quality imagery is definitely the worst offender.  Not that I'm complaining or looking for your sympathy, but lest there be any doubt as to whether your subscription dollars are getting rolled back into CalTopo development:

By way of background, while Google's Satellite layer is great, I can't do any server-side manipulation on it, whether that's generating PDFs or combining it with enhanced relief shading.  Although a secondary issue, I also can't provide offline copies of it for use by SAR teams in remote environments.

NAIP data has always been public, but in the past, acquiring it was a bit awkward, requiring the mailing of many terabytes of drives back and forth.  I tried to skirt around this by stitching the aerial backgrounds from the USGS's new "USTopo" maps into a seamless layer called USTopo Imagery.  This worked, but the quality wasn't great, and update cycles were delayed vs going directly to the source.  I've been on the hunt for some time, and as drive costs dropped, was seriously considering biting the bullet and acquiring a physical copy.

Then Andrew Johnson from Gaia GPS pointed me at the aws-naip public S3 bucket, and it was off the races.  Yes, this was expensive.  Yes, it's a roll of the dice when big-name companies like Google, MapBox and ESRI have better datasets out there.  From a business perspective, maybe it won't work out.  But I believe that high quality aerial imagery that I can repurpose as needed is strategically vital to CalTopo's future, so I decided to roll the dice and here we are.

The biggest difference between the NAIP and USTopo Imagery layers is quality.  The NAIP layer goes to zoom 17 (~1m per pixel) while USTopo Imagery went to 16 (~2m per pixel), but even at zoom 16 there's a noticeable quality difference between the two.

USTopo Imagery view of "Half Dome Village" aka Curry Village, Yosemite NP
Same location viewed using the NAIP layer

NAIP is generated in 3 year cycles, i.e. one third of the continental US is overflown each year.  Not content with a single NAIP layer, I generated two versions - one for 2011 to 2013 and one for 2013-2015.  Most places in the continental US should have two different dates available, either so that you can see how things have changed with time, or in case one revision has too much snow, shadows in the wrong place, etc.  Long term, I hope to grow the date range.

Prior imagery of the same location.  In this case, it's not much different.

NAIP is also distributed as 4-band imagery, with a near infrared channel in addition to the standard red, green and blue.  I captured this and rendered it out into a separate layer, which allows for some interesting data processing.  Right now, I'm still conflicted as to whether it's actually useful or just a neat party trick.

The Aerial Imagery section of the layer dropdown now has a "False Color IR" option.  This uses the near IR channel for red, red for blue, blue for green, and drops the green entirely.  As a result, the difference between near IR and IR is accentuated, drawing stark contrast between vegetation and manmade, dirt or rock surfaces, regardless of actual color.

False-color IR view.  No, it's not some weird 3D glasses thing.

The computed difference between red and near IR is also available as a vegetation shading option for custom MapBuilder layers, called "Infrared Reflectance".  With this option you can generate traditional looking topo maps with super-accurate vegetation shading, but as always there's a catch: areas that were shadowed in the original image show as white rather than the appropriate vegetation shading.

Custom MapBuilder layer with the IR Reflectance background.  Note the white band in the meadow at the top of the picture, which is shaded in the original image, not actually vegetation-free.
Deprecation of Existing Layers

With the layer dropdown getting increasingly complicated, this was also a good time to clean house.  The "ArcGIS Topo" option isn't as clean as my USGS map scans, but I originally included it because it covered Alaska.  That's no longer an issue, so it's gone.  USGS 1:250k maps aren of limited utility with Google Terrain and MapBuilder Topo; gone.  USTopo Imagery is inferior to the NAIP layer in pretty much every way; gone.  CA Visitor Maps had some visitor maps that can't be found elsewhere, but I need to move past state-specific layers in the dropdown, and I hope to grow the NPS and USFS visitor maps soon to help make up the gap.

All of these layers are still accessible in two ways.  First, any existing maps or links that referenced them will continue to work, although they won't display properly in the layer dropdown.  Second, they're all available as prefill custom sources.  Click on Add New Layer -> Add Custom Source, and choose the layer you want out of the "prefill with" dropdown.

Use a particular layer a lot?  Save it to your account so that it's always available.

Monday, January 2, 2017

2016 Q3/Q4 In Review

The CalTopo blog has been quiet since spring, but that doesn't mean a lack of progress, much less a lack of work.  Time to take a quick look back at the second half of 2016.

First, the personal front.  CalTopo has been my full-time job since May, and although it's averaged more than 40 hours per week, I did manage to mix a bit of vacation in.  Some of it traditionally enjoyable:

W Ridge of Pigeon Spire, Bugaboos

And some of it just grueling:

Leading the morning briefing on a campaign search after a full week of 18-20hr days.

However the universal theme for the summer, and the reason for the lack of blog updates, was that I was simply trying to stay afloat.  In between email deluges, that meant tracking down some scaling and performance issues that would always seem to trigger a crash and site outage at the most inconvenient times.

By fall, I had the performance issues sorted out, and decreasing seasonal usage lessened my customer support workload.  So it was time to get cranking on improvements.

Two lines simultaneously open for editing.
The largest of those was a major UI overhaul, moving most editing from modal, bottom-of-screen dialogs to modeless ones that stack up on the side of the screen.  This brought with it a number of improvements, including massive performance increases for large datasets, per-object visibility toggling, simultaneous editing of multiple objects (such as neighboring polygons), instant syncing of line/marker style to the map, and single-step drawing (the old UI required you to choose a style, hit OK, and then start drawing).

Per-object visibility makes it easier to clean up a cluttered map.

The other changes were much smaller but still much-needed.  Auto-drawing now has the option for larger lines (I need to make this the default), which helps prevent the oft-recurring problem of accidentally clicking next to the stream or trail rather than on it.

Auto-drawing with larger line widths.

DEM (digital elevation model) shading allowed the creation of custom shaded layers based on slope angle, elevation and aspect, but required understanding a cryptic syntax (such as s30-60e4000-6000f FF0000).  This has been replaced with a friendlier dialog that allows you to select values from dropdowns and colors from a color picker:

Preferences, such as a user's preferred datum and coordinate system, used to be stored as a browser cookie.  By storing just an ID cookie and tracking preferences in a server-side database, I can store more information without risking exceeding the cookie size limit.  This allowed me to expand to the print page, remembering a user's last selected page size, scale, and other features.

WMS and WMTS layers have been supported for a while, but reverse engineering the WMS request syntax was tricky for casual users.  The Add Custom Layer dialog now has an auto-probe option that will talk to a WMS or WMTS endpoint and try to configure the URL template automatically, making it easier to pull more third-party sources into CalTopo, particularly government run servers with a wealth of public domain (but limited geographic coverage) data.

As well as lots of smaller changes not worth listing.  That brings us to about November, when I switched gears and began working on map data instead of features.  Those are for a subsequent post, but suffice to say some exciting changes have recently gone live.