Tuesday, September 5, 2017

CalTopo Guide to Wildland Fire Information

NOTE: I wrote the bulk of this post in July, but shelved it due to some pending layer changes.  With the western US currently on fire, it seemed like a good time to finally wrap it up, although some of the references are no longer timely.

Here in Truckee, smoke from the Detwiler fire - 100 miles away - drifted in two nights ago.  A good reminder that I've been meaning to do a post on wildland fire information, using both CalTopo and other tools.  Whether there's a fire burning near your house or, like me, you're just trying to dodge smoke in order to exercise, getting accurate information can be a frustrating experience if you don't know where to look.

Not to pick too heavily on the mainstream media, but if I were trying to learn about the fire currently blowing smoke over the Sierra, the top Google News result is a recently published LA Times story:

The headline is a bit sensational - the exact quote in the article is that weather satellites spotted "explosive fire behavior" - but that's the same as it ever was.  Despite the article's many small but not terribly useful details, such as firefighters tackled 2- to 4-foot flames, the core information is wanting.  The fire perimeter map is over a day old, and their included GOES satellite image is indecipherable to anyone who doesn't know what they're looking at, myself included:

So, where I do go for information?

First stop is a set of "real time" (technically, several per day) CalTopo layers sourced from the two MODIS satellites (named Aqua and Terra) and VIIRS.  Current visible-spectrum imagery from the satellites is available through the standard layer dropdown, in both standard and relief-shaded versions.

Generally one of the satellite passes has better resolution and color quality than the others, and the current day's imagery is often not available until late in the afternoon.  Here's yesterday's Terra pass over the Detwiler fire:

As a side note, even smaller fires and controlled burns often produce smoke that's visible to the satellites.  Something to keep in mind the next time the news excitedly proclaims that a fire is now VISIBLE FROM SPACE, along with a MODIS capture like the one above.

The MODIS satellites also capture infrared imagery, which is processed and used for hot spot detection.  Generally hot spot data comes from a series of midday and overnight passes, capturing fire behavior at two points in the diurnal cycle.  These hotspot detections form the core of CalTopo's current fire activity layer:

Each dot represents a hotspot detection.  Color indicates age - red is detections less than 12 hours old, orange is 12-24 hours old, and yellow is 24-48 hour sold.

MODIS data is stitched together from a number of satellite passes, with each pass covering a long and narrow swath of the planet.  The image resolution is nominally around 1 kilometer per pixel (ie each pixel in the MODIS image represents 1 square kilometer), however the resolution drops as you move towards the edge of the pass.   As a result, some hotspots have a larger error radius than others.  Zooming in on the map shows that the error radius of each hotspot as a semi-transparent circle.

Technically, the pixels are in a rectangular grid, and each hotspot detection comes with separate X and Y error values. Due to the complexities of properly mapping this grid, I average the X and Y values into a single, circular error radius:

In the above image, the older (orange) points have a much larger error radius than the newer (red) ones.  All we can know is that there is a hotspot somewhere inside each circle; the entirety of the orange circles in particular is probably not on fire.  It's also worth noting that because the MODIS satellite is generally taking a semi oblique image, rather than looking straight down, terrain with heavy vertical relief can throw the estimated locations off - a single detection on the far side of a steep river canyon may be an erroneous interpolation and not actually indicate that the fire has crossed over.

Zooming in more shows not only the time of the detection, but the temperature, radiated power, and confidence that the detection is actually a fire.  While it can be fun to geek out on, I'm certainly not qualified to make any useful predictions based on that information.  If anyone reading this is, please chime in.

Infrared hotspot data is also available from VIIRS, and I'm looking at adding that along with other data sources such as GOES, so the accuracy of the fire activity layer may improve for next season.  Both MODIS and VIIRS hotspot detection data is available straight from the source at https://earthdata.nasa.gov/earth-observation-data/near-real-time/firms/active-fire-data.

The current fire activity layer also includes smoke information from the NOAA hazard mapping system (HMS), typically updated every several hours.

MODIS is a good rough indication of a fire's activity, but it's not perfect.  In the US, a fire's incident management team keeps track of the actual fire perimeter using a combination of aircraft observations, including nighttime IR flights, and GPS tracks recorded by teams on the ground.  These fire perimeters are typically uploaded once a day to GeoMAC, and then pulled into the CalTopo fire activity layer as crosshatched yellow polygons:

Detwiler perimeter, a little hard to see through the smoke data also included in the fire activity layer.
CalTopo also sources historic fire perimeters from GeoMAC for the fire history layer.  Wildland firefighters might scoff at the notion of calling 15 years of perimeter data "historical", since they use decades' worth of data to analyze fire behavior in a given location, but I have yet to find a better nationwide source.

Although not guaranteed, the fuel load in recently burned areas will generally be lower than in areas that haven't burned for a while.  Old fires may also be bounded by existing fuel breaks that can help stop the spread of a new fire.  Because of these factors, it can be useful to look at existing fire perimeters when trying to figure out how and where a fire might spread:

This part of the Shasta-Trinity has burned a bit
As a reminder, subscribers can pull these layers (along with all of CalTopo's other layers) into Google Earth via super ovlerays for 3-D viewing:

If you want to step outside the CalTopo bubble, one of the first questions to ask is "who is responsible for fighting the fire?"  Once a fire gets large enough this is generally available from the news, but it's also helpful to look at the CalTopo land management layer to figure out whether a fire is on federal or non-federal land.  This is a good first-order approximation, but keep in mind that fires on non-federal land could be the responsibility of either a state or local agency.  Agencies will also swap responsibility for some areas depending on who has nearby resources, so responsibility areas do not map 1:1 with land ownership.

Detwiler fire approaching, but not yet on, Forest Service land
For fires on federal land, my first stop is generally InciWeb, which has all kinds of good information:

InciWeb home page / incident list

Each incident page has a number of easy to miss tabs near the top.  The news articles (aka press releases) contain dry but substantive and useful information, and the maps tab contains links to the actual incident maps.  Generally these are updated daily as part of the incident action plan (IAP), and not only show the fire perimeter, but break it down into established fire line and uncontrolled fire edge.  Coming from a SAR background where incident data is often restricted due to law enforcement concerns, the ready availability of fire information is refreshing:

Southern edge of the Whittier fire 

If InciWeb  doesn't have the fire listed, or if you want to dig really deep into the details, my next stop is the national interagency fire center FTP site at http://ftp.nifc.gov/incident_specific_data/.  It takes a little digging (based on both location and jurisdiction) to find a specific fire, but the NIFC server generally has more detail than inciweb, and at least in California, details for state-managed fires.

Saturday, July 29, 2017

Summer Layer Update

This morning I deployed a bunch of new layer changes, some of which have been in progress for a while.  From small to large:

First up, the 40' contour layer has been redone to use MapBuilder-based contour lines.  This allows for clean, accurate contour lines even when zoomed way in, instead of the pixellated lines that used to exist when going past zoom level 16.  It also improves contour printing, since MapBuilder layers are custom-rendered for each PDF's scale; imagery-contour maps will no longer print with dense, tiny and unreadable contour lines.

Contour lines displaying cleanly atop high-resolution Google satellite imagery.
The downside is that this means a temporary loss of contour coverage for parts of Alaska, pending MapBuilder's expansion to AK, HI and parts of Canada.  The old layer is still available as a custom source, however.  Choose Add New Layer -> Add Custom Source and choose "40ft Contours" from the prefill dropdown.  Use the "save to account" link at the bottom left of the dialog to save this to your account for easy future access.

CalTopo has a lot of powerful features, but they also tend to be obscure and awkward to use.  This is particularly true of some of the custom layers, such as MODIS satellite imagery and sunlight analysis, since they require using the Add Layer dialog and going through a handful of steps to tweak a single parameter.  The hiccup to improving this has been adding configurability to the layer UI, which is now done - both base and overlay layers support dropdowns for additional configuration options.

The simplest implementation of this is slope angle shading - fixed and gradient are now condensed into a single checkbox with a selection dropdown.  Some people might consider this a step backwards, since it potentially requires two clicks rather than one.  However a consistent issue I've seen with new users is turning on both fixed and gradient shading at once, rendering the underlying map unreadable.  The rationale for collapsing them is partially to save users from that mistake and partially to save space in the ever-growing layer list.

Configurability also allowed me to move the sunlight analysis out of the "add new layer" menu and into the standard layer list.  Check the Sun Exposure layer box and you get two dropdowns - a time of year and a time of day. 

These mimic the Add New Layer -> Add Sunlight Analysis dialog, but make it far easier to walk the time-of-day selector up or down to see how sunlight and shading vary throughout the day.  This has been covered previously, but just for grins here's another example, showing the current sun exposure on Mt Whitney at 8AM:

Daily satellite imagery has been another hard-to-access feature.  I've had a "latest MODIS" layer for some time, but if the most recent exposure happened to be cloudy, dialing back the clock using the archived MODIS feature was a pain - not just having to add a custom layer, but also needing to iterate through the cycle multiple times just to find a clear day.

The "latest MODIS" layer has now been replaced with a daily satellite group that has three different layers - daytime, daytime w/ shaded relief, and nighttime.  After you choose the layer, you can quickly change either the observing satellite or the date of the capture:

Prior-year options provide two dates spaced several days apart, in order to increase the chances of finding a clear day.  The VIIRS satellite has also been added to the satellite list alongside MODIS's Aqua and Terra.

One difficulty I've had with MODIS satellite imagery is figuring out exactly what I'm looking at.  To facilitate that, the "Day (Relief)" layer automatically blends in the enhanced (aka multiply-blend) relief shading:

View of the Yosemite-area high country from a recent clear day.  The relief shading makes it easier to identify terrain features.

Although probably of little practicality, there is also a nightly capture from VIIRS:

Laugh now, but when the apocalypse hits, this will be a critical datapoint for charting the downfall of civilization.

With those layers out of the way, it's time for the big news of the day: weather mapping.  If you're like me - and I readily accept that the average American is definitely not - you've probably spent a fair bit of time with the NWS's point forecast tool.  Want to know how cold it will be overnight? One grid will be 1000' below where you want to camp, while the neighboring one is 2000' above.  Wondering how to dodge an incoming storm while still getting after it?  I'll often look up forecasts for at least a dozen places to get a sense of where the storm is tracking or how strong the rain shadow effect will be.

The best products are the ones you build to solve your own problems, and while they have yet to see much real-world validation, so far I'm pretty happy with the three new weather layers.

The temperature layer interpolates from the NWS forecast grid using CalTopo's elevation data and the 3.5 degree per thousand foot rule of thumb.  It's certainly not perfect: I have to guess at the elevation the NWS is using for each forecast grid square, there are cases where the 3.5 degree rule doesn't apply, such cold air settling in mountain valleys, and mountain weather forecasting is difficult and imprecise to begin with.  However, it still provides a useful visual of the daily highs and lows.

It's definitely summer out there, folks

The color gradient is good for a big-picture overview, but it lacks precision.  For that, the "from" and "to" dropdowns let you set minimum and maximum cutoffs.  To see everything with an overnight low of 40 or higher, set the from field to 40:

The precip layer will show forecasted precipitation for the next 24 or 48 hours, with options for both rain and snow.  While it will probably see some revisions when winter rolls around and I have additional datasets to play with - both larger and colder storms - it's still a useful big-picture indicator as is.  This February, a planned Ouray ice climbing trip was scrapped at the last minute due to unseasonably warm conditions, leaving me with a few extra days of mountain biking in the southwest while en route to Silverton.  With rainy weather rolling through, it took a while to wrap my head around the forecasts for the entire Southern UT / Northern AZ / Northwestern NM region.  A visualization like the one below would have come in useful:

Lacking such quality weather info, I found myself rained out in Moab and had to make do with a pair of backcountry days in the La Sals, which were nothing short of amazing.  So perhaps there's such a thing as over-planning and over-analyzing and we should all just go with the flow.  Except that were it not for CalTopo's planning tools, I'd probably still be trying to extricate myself from some of the dense growth we bushwhacked through.

Both of these layers have their quirks.  In the temperature screenshot above, you can see that the South side of Giraud (unshaded area at the bottom of the image) appears inexplicably colder than the North side.  In the precipitation screenshot, the storm appears to be very respectful of the New Mexico state line, presumably because each side is coming from a different forecast office.  In these situations, it's sometimes best to go straight to the source.

The "NWS Forecast Grid" layer is an interactive data layer showing the center points of the NWS forecast grid squares.  At zooms 12 and above, there is a dot for each grid square; at 11 and below, only a portion of the squares are shown.  Mousing over a grid point shows the elevation I'm using for temperature interpolation, which is just a guess based on averaging points in the grid and will precisely match the elevation the NWS uses.  Clicking on a point opens a new tab with the NWS hourly weather chart:

Going back to the Giraud example, let's take a look at the source forecast data:

Remember that the temperature is representative of the entire 2.5km grid square and not that exact point, but the North side is forecast to be warmer, at higher elevations, than the South side.  Mousing over the 47 degree point shows that I'm estimating it as 11375'; clicking on it, the NWS shows 11132'.  I estimate the 37 degree point South of that as being at 10325', and NWS says 10696.  So it's not just a quirk in my interpolation - they are forecasting the North side to be significantly warmer, despite being higher.

Is that an accurate reflection of reality?  I couldn't say, but I would guess the NWS forecast lacks that level of pinpoint precision.  But I do know that I'd rather have all the data from which to make my own informed decision, than a somewhat arbitrarily chosen point forecast or two.

Wednesday, July 19, 2017

Announcing Team Accounts

Team accounts are a new feature designed to both improve data sharing within an organization, and allow organizations to buy a single subscription covering all their members.

Maps can be saved to either an individual's account or the team account, and both sets of data are rolled together in a user's account dialog.  Because the "your maps" table is sorted chronologically, new work saved to the team account will automatically bubble up and be visible to other team members:

Icons and custom map layers saved to the team account will automatically be visible to all team members as well; if you regularly use specialized icons or custom layers, this is also a great way to push them out to team members - no more emailing around GPX files with the latest copy of your team's customizations.

Team accounts come with a dual pricing scheme - $500/yr for organizations consisting primarily of unpaid volunteers, and $2000/yr for organizations primarily made up of paid professionals.  Both versions cover organizations up to 100 people; for larger organizations, please email info@caltopo.com.  If there is interest, I may be able to add a cheaper small-business option for companies with only a handful of employees.

Beyond data sharing and customization, this price includes pro-level features for all team members, and a site license for offline use covering both team computers and personal computers owned by team members.

To sign up for a team account, go to https://caltopo.com/group/join or https://sartopo.com/group/join.  For more information see the knowledge base article on team accounts. 

Individual accounts remain unchanged.

Wednesday, July 12, 2017

SAR Grab Bag

Once again, although the blog has not been updated, the wheels of progress are still turning at CalTopo HQ.  Here's a quick run down on some new features that, while available to everyone, will mostly appeal to SAR users.

Custom Icons
CalTopo has long supported custom icons by allowing you to enter a URL on the icon chooser, but the built-in limitations made this clumsy - you had to host the image elsewhere, and re-enter the URL for each new icon.  Now, you can upload the files directly to CalTopo and tie them to your account (limited to pro-level users only):

Once uploaded and configured, your icons will appear as a new row in the icon chooser:

Variable Opacity
Lines and polygons now feature an opacity setting, so that you can create partially-transparent features that allow the underlying basemap to show through.  On the SAR side, opacity lives on operational periods and cascades down to child assignments and tracks, making it easy to show existing search efforts without completely obscuring the map.

There's a new "buffer" object type that lets you trace along a linear feature and create a polygon encompassing all points within a specified distance, making it easy to draw an assignment that actually covers everything within 200' of a trail or drainage.

After selecting New -> Buffer, you'll see a dialog asking you to select the buffer's size:

From there you'll see the normal line drawing dialog and UI.  When you finish drawing, the line gets converted to a polygon.  At the moment, this is a one-way operation: as with sectors, once a buffer is created it's just another polygon.  There's no way to alter the line that the buffer was created from.

Snap To Existing Lines and Polygons
Playing well with buffers, the auto-draw tool has been extended to let you snap to existing map objects like lines and polygons.  For maps built on non-overlapping gridded areas, like a search planning map, this makes for a much cleaner end product than trying to manually trace out two polygon boundaries to match each other.

Assignment Letters
SAR planning is still a wild west of differing approaches, but it's fairly common for teams to want to identify segments via letters, and teams assigned to those segments via numbers.  There are a number of ways SARTopo could handle this, including having separate segment and assignment objects, but I decided to go with the simple and universal approach of giving assignments two properties: a number and a letter.  You can use both, or either, and the map labels should reflect this intelligently.  The 104 form has also been updated with a space for the letter.

Assignment Summary
A feature I'm particularly excited about is the assignment summary form, accessible via the assignment bulk ops menu.

For background, common practice on searches is to prepare a separate assignment form for each team, which is also used to capture team members' names.  This is generally done using multipart carbonless copy paper; the team gets the top (most legible) copy, and the CP keeps the rest.  Capturing the names of each person on an assignment is important for safety and accountability reasons, but it's hard to print onto loose-leaf carbonless paper.  Doable, but easy to mess up - so the forms are often filled out by hand, even if some of the information is already in SARTopo.

From my involvement with large multi-day incidents, getting large stacks of assignment forms properly filled out with the right details is one of the major tasks keeping people up late and depriving them of adequate sleep.  And since teams in the field only have their own form, it limits their situational awareness.

It's also common practice to include an assignment-specific map with with each form, a feature SARTopo supports.  However, for both logistical and situational awareness reasons, I've been pushing people away from that feature and towards printing a tiled multi-page incident map to be included in the IAP - which is the way wildland fire incidents have done it for some time.

The assignment summary form extends this approach to SAR 104 forms - an approach that's also in line with the way the rest of the ICS world works, albeit in a more paper-friendly fashion.  Pertinent information for all the assignments is condensed into a page or two, suitable for inclusion in an IAP without burning through too much paper.  Carbonless forms would still be used for tracking team members' names.

As with the "print a coordinate list" feature, fields to be displayed are user-selectable.  You can print full details and a mini-map per assignment, or condense each one down to a single row.

End result:

Along with the IAP summary, there's a separate deployment summary that makes use of newly-added team size and priority fields to create a nice single-page reference for operations to use while forming and deploying teams:

Thursday, May 4, 2017

Introducing the GPSIO Extension

Most newer GPSs support USB Mass Storage, which means that when plugged into your computer, they show up as a drive with all your routes, tracks and waypoints saved as GPX files.  Older models, such as the Garmin 60csx still widely used in SAR, require a special protocol to transfer your data.

For websites like CalTopo, the only option for getting data off older Garmin GPSs, and the most convenient option for newer ones, was the Garmin Communicator Plugin (GCP).  Browsers have slowly been dropping support for the NPAPI plugin architecture on which GCP was based, but the final nail in its coffin was Garmin not only discontinuing support, but deciding to no longer issue new API keys.  Since I only had an API key for http://caltopo.com/, when a Google Chrome security change forced me to move most traffic over to https several months ago, GCP became largely useless.

That was motivation enough to wrap up a long-simmering project: the GPSIO browser extension.  Compatible with both Chrome and Firefox, GPSIO acts as a conduit between your browser and the open-source GPSBabel program, which does the actual hard work of talking to your GPS.

GPSIO is still a little rough around the edges, but you are cordially invited to take it for a spin, report any problems, or - since it's open source - fix them yourself.  Although GPSIO only supports Garmin GPSs at the moment, I would welcome code contributions to expand it to work with the wide range of GPS types that GPSBabel supports.

For more information, see http://gpsio.caltopo.com/.

Once installed, using GPSIO with CalTopo is pretty simple: under the Import or Export menu, click the "Connect via GPSIO" link.

Sunday, April 16, 2017

MapBuilder: It's Back And Better Than Ever

CalTopo has been having some performance issues lately, no point in trying to pretend otherwise.

Behind the scenes, there two different platforms: a Python/Mapnik/PostGIS stack that powers MapBuilder and auto-routing, and a Java app for everything else.  MapBuilder has been struggling for a while, but lately that started bleeding over into the rest of the site's operations - as the number of pending MapBuilder requests grew and grew, the Java side of the house (which proxies the requests) would eventually get overwhelmed and pages would stop loading.  A week ago I rolled out a new server and a new copy of the MapBuilder database with fresh OpenStreetMap updates, and everything pretty much came screeching to a halt.

This left me in a bit of a pickle.  If I rolled the deployment back, things would temporarily improve, but my production environment would be out of sync with my new development laptop, which would make it harder to debug the issues and arrive at a permanent fix.  If I didn't roll the change back, fixing the problem would be a lot easier, but it would be a rougher ride until then.  I bumped the primary EC2 instance up to a $100/day m4.16xlarge to buy some slack, rolled up my sleeves, and got to work.

What followed was a blurry week of long hours, coffee, beer, testing in production, and marathon coding sessions that produced results brilliantly functional yet barely understandable when re-examined the next day.  Kind of like being a CS undergrad during end-of-semester crunch time.

At any rate, the MapBuilder rendering stack has been almost completely rewritten, sorry for any inconveniences over the past several weeks, and I'm confident at this point that the latest round of performance issues are solidly in the rear view mirror.  At least, confident enough to roll out two new features that were waiting on this.

First, as a subscriber-only feature (all levels), CalTopo now offers Retina/HiDPI tiles for most layers.  Rather than rendering out a separate 512px tileset, my server pulls 4 tiles from the next zoom level down and rolls them up into a single tile.  As soon as you sign in to your account, the map viewer will automatically pull 512px tiles instead of the standard 256px ones.  If this is problematic for some reason (requests get routed though my server rather than direct to S3, so there are potential speed implications), it can be disabled via the config menu - look for the "Retina/HiDPI" option at the bottom of the Display group.

The other big news is the addition of a "hybrid shading" option to MapBuilder, and a new prefab layer based on it named MapBuilder Hybrid.  Hybrid shading blends the green-gradient canopy shading used by MapBuilder Topo with aerial imagery, producing a backdrop that shows you much of the detail you'd get from imagery, but with less of the visual noise that makes it hard to pick out other map features.

MapBuilder Hybrid, Tuolumne Meadows.
Zooming in, you can make out individual trees, but they don't overwhelm the image or distract from the trails and contours.

Example from the Casacades

Produces a nice visual effect at wide zoom levels, too.

Admittedly I'm both biased and a little loopy from the past week, but I have the MapBuilder Hybrid layer loaded on a 4k monitor and I can't stop staring at it.  I've been wanting to add this for a while and am psyched to get it live, although this is an early version that will probably see stylistic tweaks over time.

Tuesday, March 7, 2017

CalTopo Offline

In the very beginning, I started what would ultimately become CalTopo as a hobby project focused on offline environments - a command post or tailgate somewhere off in the woods far away from the nearest internet connection.  As CalTopo grew, that ability for entirely self-contained operation has remained as a little known but fundamental building block of the codebase.  Flip out a few of the underlying bits - Google Maps with OpenLayers, a standalone database with HSQLDB, Tomcat with Winstone - and you'd get a free-standing Java web app with no external dependencies.

For search and rescue, those who knew the secret handshake could get a similarly configured copy of SARTopo, delivered via physical thumb drives along with statewide map data.  While this system worked for a while, it's been chugging along on a donut spare and one headlight for some time.  Although releasing copies of a subscription-driven website into the wild feels a bit risky, it's past time to make the offline functionality a supported feature that doesn't require my involvement with every install.

Welcome to CalTopo (and SARTopo) Offline.

Due to the costs of delivering map data and the extra support footprint it will present, there was no way I could roll the offline functionality into the existing subscription levels.  Instead there is now a $100/yr offline subscription level that also includes all pro-level features.  First responders who have upgraded to a SAR account will see a $50/yr offline upgrade option.

After upgrading, your account dialog will have an offline access tab with links to downloading both map data and a copy of the program:

Map data is provided in 1 degree by 1 degree blocks, in MBTiles format.  Available layers include USGS 7.5' topos, forest service maps, slope angle shading, elevation data for profiles and viewsheds, and NAIP aerial imagery down to 1 meter resolution; most blocks include 2 years' worth of imagery.

Each account is allowed 300GB worth of map downloads per year, with no geographic restrictions other than being limited to the lower 48 states.   It's up to you whether you want to skip the imagery and cover a larger footprint, or get 2 years worth of 1 meter imagery for a smaller coverage area.  Coverage is available for the contiguous US:

I'd post some screenshots, but there's not much to show - the offline version looks very much like caltopo.com.  For more information on how this all works, check out some of the very thorough documentation written Patty Lindsay (there is hopefully more documentation of this caliber coming to help.caltopo.com).