Wednesday, July 19, 2017

Announcing Team Accounts

Team accounts are a new feature designed to both improve data sharing within an organization, and allow organizations to buy a single subscription covering all their members.

Maps can be saved to either an individual's account or the team account, and both sets of data are rolled together in a user's account dialog.  Because the "your maps" table is sorted chronologically, new work saved to the team account will automatically bubble up and be visible to other team members:


Icons and custom map layers saved to the team account will automatically be visible to all team members as well; if you regularly use specialized icons or custom layers, this is also a great way to push them out to team members - no more emailing around GPX files with the latest copy of your team's customizations.

Team accounts come with a dual pricing scheme - $500/yr for organizations consisting primarily of unpaid volunteers, and $2000/yr for organizations primarily made up of paid professionals.  Both versions cover organizations up to 100 people; for larger organizations, please email info@caltopo.com.  If there is interest, I may be able to add a cheaper small-business option for companies with only a handful of employees.

Beyond data sharing and customization, this price includes pro-level features for all team members, and a site license for offline use covering both team computers and personal computers owned by team members.

To sign up for a team account, go to https://caltopo.com/group/join or https://sartopo.com/group/join.  For more information see the knowledge base article on team accounts. 

Individual accounts remain unchanged.

Wednesday, July 12, 2017

SAR Grab Bag

Once again, although the blog has not been updated, the wheels of progress are still turning at CalTopo HQ.  Here's a quick run down on some new features that, while available to everyone, will mostly appeal to SAR users.

Custom Icons
CalTopo has long supported custom icons by allowing you to enter a URL on the icon chooser, but the built-in limitations made this clumsy - you had to host the image elsewhere, and re-enter the URL for each new icon.  Now, you can upload the files directly to CalTopo and tie them to your account (limited to pro-level users only):


Once uploaded and configured, your icons will appear as a new row in the icon chooser:


Variable Opacity
Lines and polygons now feature an opacity setting, so that you can create partially-transparent features that allow the underlying basemap to show through.  On the SAR side, opacity lives on operational periods and cascades down to child assignments and tracks, making it easy to show existing search efforts without completely obscuring the map.


Buffers
There's a new "buffer" object type that lets you trace along a linear feature and create a polygon encompassing all points within a specified distance, making it easy to draw an assignment that actually covers everything within 200' of a trail or drainage.


After selecting New -> Buffer, you'll see a dialog asking you to select the buffer's size:


From there you'll see the normal line drawing dialog and UI.  When you finish drawing, the line gets converted to a polygon.  At the moment, this is a one-way operation: as with sectors, once a buffer is created it's just another polygon.  There's no way to alter the line that the buffer was created from.


Snap To Existing Lines and Polygons
Playing well with buffers, the auto-draw tool has been extended to let you snap to existing map objects like lines and polygons.  For maps built on non-overlapping gridded areas, like a search planning map, this makes for a much cleaner end product than trying to manually trace out two polygon boundaries to match each other.



Assignment Letters
SAR planning is still a wild west of differing approaches, but it's fairly common for teams to want to identify segments via letters, and teams assigned to those segments via numbers.  There are a number of ways SARTopo could handle this, including having separate segment and assignment objects, but I decided to go with the simple and universal approach of giving assignments two properties: a number and a letter.  You can use both, or either, and the map labels should reflect this intelligently.  The 104 form has also been updated with a space for the letter.

Assignment Summary
A feature I'm particularly excited about is the assignment summary form, accessible via the assignment bulk ops menu.

For background, common practice on searches is to prepare a separate assignment form for each team, which is also used to capture team members' names.  This is generally done using multipart carbonless copy paper; the team gets the top (most legible) copy, and the CP keeps the rest.  Capturing the names of each person on an assignment is important for safety and accountability reasons, but it's hard to print onto loose-leaf carbonless paper.  Doable, but easy to mess up - so the forms are often filled out by hand, even if some of the information is already in SARTopo.

From my involvement with large multi-day incidents, getting large stacks of assignment forms properly filled out with the right details is one of the major tasks keeping people up late and depriving them of adequate sleep.  And since teams in the field only have their own form, it limits their situational awareness.

It's also common practice to include an assignment-specific map with with each form, a feature SARTopo supports.  However, for both logistical and situational awareness reasons, I've been pushing people away from that feature and towards printing a tiled multi-page incident map to be included in the IAP - which is the way wildland fire incidents have done it for some time.

The assignment summary form extends this approach to SAR 104 forms - an approach that's also in line with the way the rest of the ICS world works, albeit in a more paper-friendly fashion.  Pertinent information for all the assignments is condensed into a page or two, suitable for inclusion in an IAP without burning through too much paper.  Carbonless forms would still be used for tracking team members' names.


As with the "print a coordinate list" feature, fields to be displayed are user-selectable.  You can print full details and a mini-map per assignment, or condense each one down to a single row.

End result:


Along with the IAP summary, there's a separate deployment summary that makes use of newly-added team size and priority fields to create a nice single-page reference for operations to use while forming and deploying teams:


Thursday, May 4, 2017

Introducing the GPSIO Extension

Most newer GPSs support USB Mass Storage, which means that when plugged into your computer, they show up as a drive with all your routes, tracks and waypoints saved as GPX files.  Older models, such as the Garmin 60csx still widely used in SAR, require a special protocol to transfer your data.

For websites like CalTopo, the only option for getting data off older Garmin GPSs, and the most convenient option for newer ones, was the Garmin Communicator Plugin (GCP).  Browsers have slowly been dropping support for the NPAPI plugin architecture on which GCP was based, but the final nail in its coffin was Garmin not only discontinuing support, but deciding to no longer issue new API keys.  Since I only had an API key for http://caltopo.com/, when a Google Chrome security change forced me to move most traffic over to https several months ago, GCP became largely useless.

That was motivation enough to wrap up a long-simmering project: the GPSIO browser extension.  Compatible with both Chrome and Firefox, GPSIO acts as a conduit between your browser and the open-source GPSBabel program, which does the actual hard work of talking to your GPS.

GPSIO is still a little rough around the edges, but you are cordially invited to take it for a spin, report any problems, or - since it's open source - fix them yourself.  Although GPSIO only supports Garmin GPSs at the moment, I would welcome code contributions to expand it to work with the wide range of GPS types that GPSBabel supports.

For more information, see http://gpsio.caltopo.com/.

Once installed, using GPSIO with CalTopo is pretty simple: under the Import or Export menu, click the "Connect via GPSIO" link.


Sunday, April 16, 2017

MapBuilder: It's Back And Better Than Ever

CalTopo has been having some performance issues lately, no point in trying to pretend otherwise.

Behind the scenes, there two different platforms: a Python/Mapnik/PostGIS stack that powers MapBuilder and auto-routing, and a Java app for everything else.  MapBuilder has been struggling for a while, but lately that started bleeding over into the rest of the site's operations - as the number of pending MapBuilder requests grew and grew, the Java side of the house (which proxies the requests) would eventually get overwhelmed and pages would stop loading.  A week ago I rolled out a new server and a new copy of the MapBuilder database with fresh OpenStreetMap updates, and everything pretty much came screeching to a halt.

This left me in a bit of a pickle.  If I rolled the deployment back, things would temporarily improve, but my production environment would be out of sync with my new development laptop, which would make it harder to debug the issues and arrive at a permanent fix.  If I didn't roll the change back, fixing the problem would be a lot easier, but it would be a rougher ride until then.  I bumped the primary EC2 instance up to a $100/day m4.16xlarge to buy some slack, rolled up my sleeves, and got to work.

What followed was a blurry week of long hours, coffee, beer, testing in production, and marathon coding sessions that produced results brilliantly functional yet barely understandable when re-examined the next day.  Kind of like being a CS undergrad during end-of-semester crunch time.

At any rate, the MapBuilder rendering stack has been almost completely rewritten, sorry for any inconveniences over the past several weeks, and I'm confident at this point that the latest round of performance issues are solidly in the rear view mirror.  At least, confident enough to roll out two new features that were waiting on this.

First, as a subscriber-only feature (all levels), CalTopo now offers Retina/HiDPI tiles for most layers.  Rather than rendering out a separate 512px tileset, my server pulls 4 tiles from the next zoom level down and rolls them up into a single tile.  As soon as you sign in to your account, the map viewer will automatically pull 512px tiles instead of the standard 256px ones.  If this is problematic for some reason (requests get routed though my server rather than direct to S3, so there are potential speed implications), it can be disabled via the config menu - look for the "Retina/HiDPI" option at the bottom of the Display group.

The other big news is the addition of a "hybrid shading" option to MapBuilder, and a new prefab layer based on it named MapBuilder Hybrid.  Hybrid shading blends the green-gradient canopy shading used by MapBuilder Topo with aerial imagery, producing a backdrop that shows you much of the detail you'd get from imagery, but with less of the visual noise that makes it hard to pick out other map features.

MapBuilder Hybrid, Tuolumne Meadows.
Zooming in, you can make out individual trees, but they don't overwhelm the image or distract from the trails and contours.

Example from the Casacades

Produces a nice visual effect at wide zoom levels, too.

Admittedly I'm both biased and a little loopy from the past week, but I have the MapBuilder Hybrid layer loaded on a 4k monitor and I can't stop staring at it.  I've been wanting to add this for a while and am psyched to get it live, although this is an early version that will probably see stylistic tweaks over time.

Tuesday, March 7, 2017

CalTopo Offline

In the very beginning, I started what would ultimately become CalTopo as a hobby project focused on offline environments - a command post or tailgate somewhere off in the woods far away from the nearest internet connection.  As CalTopo grew, that ability for entirely self-contained operation has remained as a little known but fundamental building block of the codebase.  Flip out a few of the underlying bits - Google Maps with OpenLayers, a standalone database with HSQLDB, Tomcat with Winstone - and you'd get a free-standing Java web app with no external dependencies.

For search and rescue, those who knew the secret handshake could get a similarly configured copy of SARTopo, delivered via physical thumb drives along with statewide map data.  While this system worked for a while, it's been chugging along on a donut spare and one headlight for some time.  Although releasing copies of a subscription-driven website into the wild feels a bit risky, it's past time to make the offline functionality a supported feature that doesn't require my involvement with every install.

Welcome to CalTopo (and SARTopo) Offline.

Due to the costs of delivering map data and the extra support footprint it will present, there was no way I could roll the offline functionality into the existing subscription levels.  Instead there is now a $100/yr offline subscription level that also includes all pro-level features.  First responders who have upgraded to a SAR account will see a $50/yr offline upgrade option.


After upgrading, your account dialog will have an offline access tab with links to downloading both map data and a copy of the program:


Map data is provided in 1 degree by 1 degree blocks, in MBTiles format.  Available layers include USGS 7.5' topos, forest service maps, slope angle shading, elevation data for profiles and viewsheds, and NAIP aerial imagery down to 1 meter resolution; most blocks include 2 years' worth of imagery.

Each account is allowed 300GB worth of map downloads per year, with no geographic restrictions other than being limited to the lower 48 states.   It's up to you whether you want to skip the imagery and cover a larger footprint, or get 2 years worth of 1 meter imagery for a smaller coverage area.  Coverage is available for the contiguous US:


I'd post some screenshots, but there's not much to show - the offline version looks very much like caltopo.com.  For more information on how this all works, check out some of the very thorough documentation written Patty Lindsay (there is hopefully more documentation of this caliber coming to help.caltopo.com).

Saturday, January 7, 2017

New Year, New Layers

In my last blog post, I mentioned that I spent the tail end of 2016 cooking up some new map layers.  Well, here we go.

Terrain Shading and Custom Relief

In SAR, it's not uncommon to have a group of people huddled around a large-format map, looking at it from all angles.  While relief shading helps terrain features pop, it only works when the map is viewed from the bottom - standing at the top will cause features to invert with peaks looking like valleys.

One alternative is terrain shading (for lack of a better word), where the relief is generated using a number of different light sources, rather a single 315 degree (NW) angle.  Because of the multiple angles used, there is no "up" or "down", and the map can be viewed from any angle without playing tricks on the eye.  The downside is that in some areas it can be hard for the eye to quickly tell up from down.


For CalTopo's terrain shading, I tried to balance these tradeoffs by using 6 evenly spaced light sources plus an additional one at 315 degrees, proving a slight amount of orientation.

CalTopo enhanced relief, which uses a multiply blend filter but retains the standard
315 degree light source


CalTopo terrain relief, which uses a 7 different light sources
If the out-of-the-box terrain shading doesn't quite work for you, there's now an "Add Custom Relief" option under the Add New Layer menu.  Pick an azimuth (compass direction) and zenith (angle above the horizon) and click "Add Lighting".  Mix and match multiple combinations to reach your desired effect.



FSTopo 2016

The FSTopo maps behind the "US Forest Service" map layer have been seeing regular updates, but it's not all forward progress.  While the newer maps gained vegetation shading and better road/trail data, the delineation between public and private property moved from light, transparent gray shading to a heavy-handed gray that completely obscured all vegetation shading.  I wasn't happy with the way this looked, and wasn't happy dropping public/private land boundaries, so I did things the CalTopo way: create a new layer.  And thus the USFS 2016 layer was born.

For most purposes the now-renamed "FSTopo (2013)" and "FSTopo (2016)" layers will be interchangeable.  The old layer is better for locating land boundaries, and its white background works well for blending with aerial imagery or slope angle shading.  With its vegetation shading, the new layer is probably better suited for standalone use.

The same map as above, but this time using the "FSTopo (2016)" layer.

Expanded Alaska DEM Coverage

While not a new layer per se, I've been slowly expanding CalTopo's Alaska coverage.  Although the national elevation dataset (NED) still only covers a portion of the state, that portion has been growing, and it's time to catch up.  The NED isn't a first-class layer, but a powers a lot of other ones, including normal relief, enhanced relief, 40' contours, fixed and gradient slope angle shading, custom DEM shading, and cursor point elevations.

The current state of NED-based layer coverage in Alaska.
Note: I'm still patching up a few small errors in the dataset, but didn't want to hold this announcement up until those were fixed.

New NAIP Imagery Layer

The new layers based on the National Agriculture Imagery Program (NAIP) dataset are probably worthy of a blog post all their own.  But before I get into the good stuff, know that all this layer creation doesn't come cheap, and high quality imagery is definitely the worst offender.  Not that I'm complaining or looking for your sympathy, but lest there be any doubt as to whether your subscription dollars are getting rolled back into CalTopo development:


By way of background, while Google's Satellite layer is great, I can't do any server-side manipulation on it, whether that's generating PDFs or combining it with enhanced relief shading.  Although a secondary issue, I also can't provide offline copies of it for use by SAR teams in remote environments.

NAIP data has always been public, but in the past, acquiring it was a bit awkward, requiring the mailing of many terabytes of drives back and forth.  I tried to skirt around this by stitching the aerial backgrounds from the USGS's new "USTopo" maps into a seamless layer called USTopo Imagery.  This worked, but the quality wasn't great, and update cycles were delayed vs going directly to the source.  I've been on the hunt for some time, and as drive costs dropped, was seriously considering biting the bullet and acquiring a physical copy.

Then Andrew Johnson from Gaia GPS pointed me at the aws-naip public S3 bucket, and it was off the races.  Yes, this was expensive.  Yes, it's a roll of the dice when big-name companies like Google, MapBox and ESRI have better datasets out there.  From a business perspective, maybe it won't work out.  But I believe that high quality aerial imagery that I can repurpose as needed is strategically vital to CalTopo's future, so I decided to roll the dice and here we are.

The biggest difference between the NAIP and USTopo Imagery layers is quality.  The NAIP layer goes to zoom 17 (~1m per pixel) while USTopo Imagery went to 16 (~2m per pixel), but even at zoom 16 there's a noticeable quality difference between the two.

USTopo Imagery view of "Half Dome Village" aka Curry Village, Yosemite NP
Same location viewed using the NAIP layer

NAIP is generated in 3 year cycles, i.e. one third of the continental US is overflown each year.  Not content with a single NAIP layer, I generated two versions - one for 2011 to 2013 and one for 2013-2015.  Most places in the continental US should have two different dates available, either so that you can see how things have changed with time, or in case one revision has too much snow, shadows in the wrong place, etc.  Long term, I hope to grow the date range.

Prior imagery of the same location.  In this case, it's not much different.

NAIP is also distributed as 4-band imagery, with a near infrared channel in addition to the standard red, green and blue.  I captured this and rendered it out into a separate layer, which allows for some interesting data processing.  Right now, I'm still conflicted as to whether it's actually useful or just a neat party trick.

The Aerial Imagery section of the layer dropdown now has a "False Color IR" option.  This uses the near IR channel for red, red for blue, blue for green, and drops the green entirely.  As a result, the difference between near IR and IR is accentuated, drawing stark contrast between vegetation and manmade, dirt or rock surfaces, regardless of actual color.

False-color IR view.  No, it's not some weird 3D glasses thing.

The computed difference between red and near IR is also available as a vegetation shading option for custom MapBuilder layers, called "Infrared Reflectance".  With this option you can generate traditional looking topo maps with super-accurate vegetation shading, but as always there's a catch: areas that were shadowed in the original image show as white rather than the appropriate vegetation shading.

Custom MapBuilder layer with the IR Reflectance background.  Note the white band in the meadow at the top of the picture, which is shaded in the original image, not actually vegetation-free.
Deprecation of Existing Layers

With the layer dropdown getting increasingly complicated, this was also a good time to clean house.  The "ArcGIS Topo" option isn't as clean as my USGS map scans, but I originally included it because it covered Alaska.  That's no longer an issue, so it's gone.  USGS 1:250k maps aren of limited utility with Google Terrain and MapBuilder Topo; gone.  USTopo Imagery is inferior to the NAIP layer in pretty much every way; gone.  CA Visitor Maps had some visitor maps that can't be found elsewhere, but I need to move past state-specific layers in the dropdown, and I hope to grow the NPS and USFS visitor maps soon to help make up the gap.

All of these layers are still accessible in two ways.  First, any existing maps or links that referenced them will continue to work, although they won't display properly in the layer dropdown.  Second, they're all available as prefill custom sources.  Click on Add New Layer -> Add Custom Source, and choose the layer you want out of the "prefill with" dropdown.

Use a particular layer a lot?  Save it to your account so that it's always available.

Monday, January 2, 2017

2016 Q3/Q4 In Review

The CalTopo blog has been quiet since spring, but that doesn't mean a lack of progress, much less a lack of work.  Time to take a quick look back at the second half of 2016.

First, the personal front.  CalTopo has been my full-time job since May, and although it's averaged more than 40 hours per week, I did manage to mix a bit of vacation in.  Some of it traditionally enjoyable:

W Ridge of Pigeon Spire, Bugaboos

And some of it just grueling:

Leading the morning briefing on a campaign search after a full week of 18-20hr days.

However the universal theme for the summer, and the reason for the lack of blog updates, was that I was simply trying to stay afloat.  In between email deluges, that meant tracking down some scaling and performance issues that would always seem to trigger a crash and site outage at the most inconvenient times.

By fall, I had the performance issues sorted out, and decreasing seasonal usage lessened my customer support workload.  So it was time to get cranking on improvements.

Two lines simultaneously open for editing.
The largest of those was a major UI overhaul, moving most editing from modal, bottom-of-screen dialogs to modeless ones that stack up on the side of the screen.  This brought with it a number of improvements, including massive performance increases for large datasets, per-object visibility toggling, simultaneous editing of multiple objects (such as neighboring polygons), instant syncing of line/marker style to the map, and single-step drawing (the old UI required you to choose a style, hit OK, and then start drawing).

Per-object visibility makes it easier to clean up a cluttered map.

The other changes were much smaller but still much-needed.  Auto-drawing now has the option for larger lines (I need to make this the default), which helps prevent the oft-recurring problem of accidentally clicking next to the stream or trail rather than on it.

Auto-drawing with larger line widths.

DEM (digital elevation model) shading allowed the creation of custom shaded layers based on slope angle, elevation and aspect, but required understanding a cryptic syntax (such as s30-60e4000-6000f FF0000).  This has been replaced with a friendlier dialog that allows you to select values from dropdowns and colors from a color picker:


Preferences, such as a user's preferred datum and coordinate system, used to be stored as a browser cookie.  By storing just an ID cookie and tracking preferences in a server-side database, I can store more information without risking exceeding the cookie size limit.  This allowed me to expand to the print page, remembering a user's last selected page size, scale, and other features.

WMS and WMTS layers have been supported for a while, but reverse engineering the WMS request syntax was tricky for casual users.  The Add Custom Layer dialog now has an auto-probe option that will talk to a WMS or WMTS endpoint and try to configure the URL template automatically, making it easier to pull more third-party sources into CalTopo, particularly government run servers with a wealth of public domain (but limited geographic coverage) data.

As well as lots of smaller changes not worth listing.  That brings us to about November, when I switched gears and began working on map data instead of features.  Those are for a subsequent post, but suffice to say some exciting changes have recently gone live.