Friday, October 13, 2017

Wind Forecast Layer

When I added the weather forecast layer this summer, I mentioned my frustrations with the way the NWS requires you to bring up endless point forecasts to get an accurate picture of local precipitation amounts.  With the Northern California fires, I'm running into the same issue with wind - I'll read about a forecasted red flag warning, but the Napa and Santa Rosa forecasts barely hit double digits.  I'll admit that I had no idea wind speeds were so locally variant - or at least, no idea that the NWS forecast grid captured such small-scale variations.

24hr wind gust plot.  From 15mph in St Helena to 50mph on Mount Hood just 3 miles away.

Fortunately I already had wind speed mapping on the back burner - it didn't get deployed along with the earlier temperature and precipitation work, but most of that code was reusable.  I can't show wind direction - even if I were to render directional arrows on the map, frequent direction changes make it impossible to show a single meaningful 24hr or 36hr wind direction.  However peak forecasted wind speeds and gusts are shown for 1hr, 6hr, 12hr, 24hr and 36hr intervals, using the same green-yellow-orange-red-purple-blue-black gradient scale as the temperature layer.

One option that does provide wind direction is the crowd favorite windy.com.  However, windy does not reflect the small-scale variations in the NWS forecast grid (see below).  As with temperature and precipitation, it's an open question as to how accurate the NWS grid variations are, but I like to provide as much raw data as possible and let users draw their own conclusions.

Windy.com plot of the same location
The wind layer is a checkbox option alongside temp and precip:


The forecast grid option immediately below will also show point speeds (remember, the point is simply the center of a grid square, and the forecast applies to the entire square), and clicking on a point will bring up the hourly weather chart.


I feel like I've pretty much run through the backlog of fire-related items I had sitting around, so I think this will be the last major layer change in response to the Northern California fires.

Update: there are now two wind layers, the "max wind speed" layer as described above, and a "wind plot" that provides forecasted directions and speeds for specific points in times, at 3 hour intervals to 12 hours, and then 6 hour intervals to 36 hours.  The same color chart is used for speed, with short lines tracing direction.  The length of the lines has no meaning.


High Resolution Post-Fire Imagery

Thanks to a generous offer from a contact on the Google Crisis Response team, CalTopo is temporarily displaying high-resolution imagery for portions of the fire-affected areas in Napa and Sonoma counties.  This imagery was taken by DigitalGlobe on Wed the 11th, and although the fires are still ongoing, it provides some degree of insight into what had burned by that point.  While there is a lot of smoke present, in many areas it's clear enough for a house-by-house assessment.

Because the original imagery is false-color infrared (which gives vegetation a strong red tint), I'm converting it to black-and-white to avoid any possible confusion between the red tint and currently burning areas.  I'm told that true-color imagery will be available at some point on Fri, and hopefully I'll be able to upgrade the layer to match.

Available imagery footprints, with fire outlines shown in red.

Step 1: Go to https://caltopo.com/l/16PD

Step 2: Enter your address in the search field at the top of the page and click GO:


Step 3: Use the zoom control at the top left of the map to zoom in:


Step 4: Mouse over the layer selector at the top right (it should read "Hybrid +1"), and in the window that comes up, adjust the "NorCal Fire" opacity slider to expose the pre-fire imagery:


It may also be helpful to temporarily turn on the "Current Fire Activity" checkbox in order to see the fire perimeters:


I've put some temporary infrastructure in place to help handle this project, but if the site is impacted too much, I may need to make some changes.  The shortlink at the top of this article (https://caltopo.com/l/16PD) will always point to the right place - if you share a URL, please share that.

Wednesday, October 11, 2017

NorCal Fire Response

The quick summary:

  • If you are seeking information on the Northern California wildfires, or trying to understand CalTopo's current fire activity layer, please see this blog post.
  • Edit: big thanks to the Google crisis response team for rapidly lifting the map quota.  The OpenLayers warning no longer applies.  CalTopo will be intermittently running on OpenLayers instead of Google Maps.  If the zoom controls at the top left of the map viewer are blue instead of white, Google layers (map, terrain, satellite) will be unavailable, and text based location search will not work.  You can use the search bar to locate lat/long or UTM coordinates, but not named places.
  • The current fire activity layer now includes VIIRS 375m detections as well as MODIS.
  • CalTopo now sports a new mobile UI.  If you have issues with it, please let me know.
The details: in case you've been living under a rock since Sunday night, California is on fire:

Sunday night's VIIRS capture: fires that look like cities

On Tue morning, CalTopo started seeing increased visits from people looking at the fire activity layer, and pageviews rapidly grew to well above normal.  This caused some scaling issues, but barring another order of magnitude in growth, those are temporarily solved with a combination of software fixes and throwing hardware at the problem.


The larger issue is that I ran into the 100k pageview limit for the Google Maps standard plan, and CalTopo stopped loading.  I've fixed this by switching to OpenLayers, which is the map viewer used in CalTopo Offline, but it's not a perfect substitute: in addition to missing Google's map layers, text-based location searches (like "santa rosa, ca") will no longer work.  The version of OpenLayers I'm using is also old and heavily customized, so there may be some compatibility issues with touch gestures on mobile devices.

I'm working with Google on fixing the issue, but in the meantime, I'll turn on Google Maps at some point in the morning, and leave it on until the quota runs out.  If the zoom +/- buttons at the top left of your screen are white, you're using the Google Maps viewer and everything should function normally.  If they're blue, it means you're using OpenLayers.

In order to better serve people looking for fire information, I've also held a marathon coding session to wrap up and deploy two back-burner upgrades.  The first is the addition of VIIRS 375m fire detections alongside MODIS.  At more than double MODIS's resolution, VIIRS should provide a more accurate picture of the fires' behavior.  Although VIIRS suffers from a limited number of passes (just like MODIS), adding a second source will increase the layer's all-important update frequency.  And certainly the most common question I've gotten today is "when will the layer update"?

Looks just like MODIS, but with more points

The current fire activity layer was already customizable (smoke vs no smoke); now you can choose all sources (a mix of MODIS and VIIRS on the same map) or view them one at a time for a cleaner image.


The second change is a revamped mobile UI.  Normally I'd hang on to a change like this for another month or two in order to test it thoroughly and smooth out the edges.  However, the vast majority - > 90% - of people seeking fire information are doing so on their phones.  And, the previous mobile UI was non-functional enough that I'm unlikely to make things worse.  I know it's 2017 already, but now the CalTopo mobile site supports 90% of the features the desktop site does.  More details in a later post once I get things cleaned up, but in the meantime let me know of any bugs.


Tuesday, September 5, 2017

CalTopo Guide to Wildland Fire Information

NOTE: I wrote the bulk of this post in July, but shelved it due to some pending layer changes.  With the western US currently on fire, it seemed like a good time to finally wrap it up, although some of the references are no longer timely.

Here in Truckee, smoke from the Detwiler fire - 100 miles away - drifted in two nights ago.  A good reminder that I've been meaning to do a post on wildland fire information, using both CalTopo and other tools.  Whether there's a fire burning near your house or, like me, you're just trying to dodge smoke in order to exercise, getting accurate information can be a frustrating experience if you don't know where to look.

Not to pick too heavily on the mainstream media, but if I were trying to learn about the fire currently blowing smoke over the Sierra, the top Google News result is a recently published LA Times story:


The headline is a bit sensational - the exact quote in the article is that weather satellites spotted "explosive fire behavior" - but that's the same as it ever was.  Despite the article's many small but not terribly useful details, such as firefighters tackled 2- to 4-foot flames, the core information is wanting.  The fire perimeter map is over a day old, and their included GOES satellite image is indecipherable to anyone who doesn't know what they're looking at, myself included:


So, where I do go for information?

First stop is a set of "real time" (technically, several per day) CalTopo layers sourced from the two MODIS satellites (named Aqua and Terra) and VIIRS.  Current visible-spectrum imagery from the satellites is available through the standard layer dropdown, in both standard and relief-shaded versions.


Generally one of the satellite passes has better resolution and color quality than the others, and the current day's imagery is often not available until late in the afternoon.  Here's yesterday's Terra pass over the Detwiler fire:


As a side note, even smaller fires and controlled burns often produce smoke that's visible to the satellites.  Something to keep in mind the next time the news excitedly proclaims that a fire is now VISIBLE FROM SPACE, along with a MODIS capture like the one above.

The MODIS satellites also capture infrared imagery, which is processed and used for hot spot detection.  Generally hot spot data comes from a series of midday and overnight passes, capturing fire behavior at two points in the diurnal cycle.  These hotspot detections form the core of CalTopo's current fire activity layer:


Each dot represents a hotspot detection.  Color indicates age - red is detections less than 12 hours old, orange is 12-24 hours old, and yellow is 24-48 hour sold.

MODIS data is stitched together from a number of satellite passes, with each pass covering a long and narrow swath of the planet.  The image resolution is nominally around 1 kilometer per pixel (ie each pixel in the MODIS image represents 1 square kilometer), however the resolution drops as you move towards the edge of the pass.   As a result, some hotspots have a larger error radius than others.  Zooming in on the map shows that the error radius of each hotspot as a semi-transparent circle.

Technically, the pixels are in a rectangular grid, and each hotspot detection comes with separate X and Y error values. Due to the complexities of properly mapping this grid, I average the X and Y values into a single, circular error radius:


In the above image, the older (orange) points have a much larger error radius than the newer (red) ones.  All we can know is that there is a hotspot somewhere inside each circle; the entirety of the orange circles in particular is probably not on fire.  It's also worth noting that because the MODIS satellite is generally taking a semi oblique image, rather than looking straight down, terrain with heavy vertical relief can throw the estimated locations off - a single detection on the far side of a steep river canyon may be an erroneous interpolation and not actually indicate that the fire has crossed over.

Zooming in more shows not only the time of the detection, but the temperature, radiated power, and confidence that the detection is actually a fire.  While it can be fun to geek out on, I'm certainly not qualified to make any useful predictions based on that information.  If anyone reading this is, please chime in.

Infrared hotspot data is also available from VIIRS, and I'm looking at adding that along with other data sources such as GOES, so the accuracy of the fire activity layer may improve for next season.  Both MODIS and VIIRS hotspot detection data is available straight from the source at https://earthdata.nasa.gov/earth-observation-data/near-real-time/firms/active-fire-data.

The current fire activity layer also includes smoke information from the NOAA hazard mapping system (HMS), typically updated every several hours.



MODIS is a good rough indication of a fire's activity, but it's not perfect.  In the US, a fire's incident management team keeps track of the actual fire perimeter using a combination of aircraft observations, including nighttime IR flights, and GPS tracks recorded by teams on the ground.  These fire perimeters are typically uploaded once a day to GeoMAC, and then pulled into the CalTopo fire activity layer as crosshatched yellow polygons:

Detwiler perimeter, a little hard to see through the smoke data also included in the fire activity layer.
CalTopo also sources historic fire perimeters from GeoMAC for the fire history layer.  Wildland firefighters might scoff at the notion of calling 15 years of perimeter data "historical", since they use decades' worth of data to analyze fire behavior in a given location, but I have yet to find a better nationwide source.

Although not guaranteed, the fuel load in recently burned areas will generally be lower than in areas that haven't burned for a while.  Old fires may also be bounded by existing fuel breaks that can help stop the spread of a new fire.  Because of these factors, it can be useful to look at existing fire perimeters when trying to figure out how and where a fire might spread:

This part of the Shasta-Trinity has burned a bit
As a reminder, subscribers can pull these layers (along with all of CalTopo's other layers) into Google Earth via super ovlerays for 3-D viewing:


If you want to step outside the CalTopo bubble, one of the first questions to ask is "who is responsible for fighting the fire?"  Once a fire gets large enough this is generally available from the news, but it's also helpful to look at the CalTopo land management layer to figure out whether a fire is on federal or non-federal land.  This is a good first-order approximation, but keep in mind that fires on non-federal land could be the responsibility of either a state or local agency.  Agencies will also swap responsibility for some areas depending on who has nearby resources, so responsibility areas do not map 1:1 with land ownership.

Detwiler fire approaching, but not yet on, Forest Service land
For fires on federal land, my first stop is generally InciWeb, which has all kinds of good information:

InciWeb home page / incident list

Each incident page has a number of easy to miss tabs near the top.  The news articles (aka press releases) contain dry but substantive and useful information, and the maps tab contains links to the actual incident maps.  Generally these are updated daily as part of the incident action plan (IAP), and not only show the fire perimeter, but break it down into established fire line and uncontrolled fire edge.  Coming from a SAR background where incident data is often restricted due to law enforcement concerns, the ready availability of fire information is refreshing:

Southern edge of the Whittier fire 

If InciWeb  doesn't have the fire listed, or if you want to dig really deep into the details, my next stop is the national interagency fire center FTP site at http://ftp.nifc.gov/incident_specific_data/.  It takes a little digging (based on both location and jurisdiction) to find a specific fire, but the NIFC server generally has more detail than inciweb, and at least in California, details for state-managed fires.

Saturday, July 29, 2017

Summer Layer Update

This morning I deployed a bunch of new layer changes, some of which have been in progress for a while.  From small to large:

First up, the 40' contour layer has been redone to use MapBuilder-based contour lines.  This allows for clean, accurate contour lines even when zoomed way in, instead of the pixellated lines that used to exist when going past zoom level 16.  It also improves contour printing, since MapBuilder layers are custom-rendered for each PDF's scale; imagery-contour maps will no longer print with dense, tiny and unreadable contour lines.

Contour lines displaying cleanly atop high-resolution Google satellite imagery.
The downside is that this means a temporary loss of contour coverage for parts of Alaska, pending MapBuilder's expansion to AK, HI and parts of Canada.  The old layer is still available as a custom source, however.  Choose Add New Layer -> Add Custom Source and choose "40ft Contours" from the prefill dropdown.  Use the "save to account" link at the bottom left of the dialog to save this to your account for easy future access.

CalTopo has a lot of powerful features, but they also tend to be obscure and awkward to use.  This is particularly true of some of the custom layers, such as MODIS satellite imagery and sunlight analysis, since they require using the Add Layer dialog and going through a handful of steps to tweak a single parameter.  The hiccup to improving this has been adding configurability to the layer UI, which is now done - both base and overlay layers support dropdowns for additional configuration options.

The simplest implementation of this is slope angle shading - fixed and gradient are now condensed into a single checkbox with a selection dropdown.  Some people might consider this a step backwards, since it potentially requires two clicks rather than one.  However a consistent issue I've seen with new users is turning on both fixed and gradient shading at once, rendering the underlying map unreadable.  The rationale for collapsing them is partially to save users from that mistake and partially to save space in the ever-growing layer list.


Configurability also allowed me to move the sunlight analysis out of the "add new layer" menu and into the standard layer list.  Check the Sun Exposure layer box and you get two dropdowns - a time of year and a time of day. 


These mimic the Add New Layer -> Add Sunlight Analysis dialog, but make it far easier to walk the time-of-day selector up or down to see how sunlight and shading vary throughout the day.  This has been covered previously, but just for grins here's another example, showing the current sun exposure on Mt Whitney at 8AM:


Daily satellite imagery has been another hard-to-access feature.  I've had a "latest MODIS" layer for some time, but if the most recent exposure happened to be cloudy, dialing back the clock using the archived MODIS feature was a pain - not just having to add a custom layer, but also needing to iterate through the cycle multiple times just to find a clear day.


The "latest MODIS" layer has now been replaced with a daily satellite group that has three different layers - daytime, daytime w/ shaded relief, and nighttime.  After you choose the layer, you can quickly change either the observing satellite or the date of the capture:


Prior-year options provide two dates spaced several days apart, in order to increase the chances of finding a clear day.  The VIIRS satellite has also been added to the satellite list alongside MODIS's Aqua and Terra.

One difficulty I've had with MODIS satellite imagery is figuring out exactly what I'm looking at.  To facilitate that, the "Day (Relief)" layer automatically blends in the enhanced (aka multiply-blend) relief shading:

View of the Yosemite-area high country from a recent clear day.  The relief shading makes it easier to identify terrain features.

Although probably of little practicality, there is also a nightly capture from VIIRS:

Laugh now, but when the apocalypse hits, this will be a critical datapoint for charting the downfall of civilization.

With those layers out of the way, it's time for the big news of the day: weather mapping.  If you're like me - and I readily accept that the average American is definitely not - you've probably spent a fair bit of time with the NWS's point forecast tool.  Want to know how cold it will be overnight? One grid will be 1000' below where you want to camp, while the neighboring one is 2000' above.  Wondering how to dodge an incoming storm while still getting after it?  I'll often look up forecasts for at least a dozen places to get a sense of where the storm is tracking or how strong the rain shadow effect will be.

The best products are the ones you build to solve your own problems, and while they have yet to see much real-world validation, so far I'm pretty happy with the three new weather layers.


The temperature layer interpolates from the NWS forecast grid using CalTopo's elevation data and the 3.5 degree per thousand foot rule of thumb.  It's certainly not perfect: I have to guess at the elevation the NWS is using for each forecast grid square, there are cases where the 3.5 degree rule doesn't apply, such cold air settling in mountain valleys, and mountain weather forecasting is difficult and imprecise to begin with.  However, it still provides a useful visual of the daily highs and lows.

It's definitely summer out there, folks

The color gradient is good for a big-picture overview, but it lacks precision.  For that, the "from" and "to" dropdowns let you set minimum and maximum cutoffs.  To see everything with an overnight low of 40 or higher, set the from field to 40:


The precip layer will show forecasted precipitation for the next 24 or 48 hours, with options for both rain and snow.  While it will probably see some revisions when winter rolls around and I have additional datasets to play with - both larger and colder storms - it's still a useful big-picture indicator as is.  This February, a planned Ouray ice climbing trip was scrapped at the last minute due to unseasonably warm conditions, leaving me with a few extra days of mountain biking in the southwest while en route to Silverton.  With rainy weather rolling through, it took a while to wrap my head around the forecasts for the entire Southern UT / Northern AZ / Northwestern NM region.  A visualization like the one below would have come in useful:


Lacking such quality weather info, I found myself rained out in Moab and had to make do with a pair of backcountry days in the La Sals, which were nothing short of amazing.  So perhaps there's such a thing as over-planning and over-analyzing and we should all just go with the flow.  Except that were it not for CalTopo's planning tools, I'd probably still be trying to extricate myself from some of the dense growth we bushwhacked through.


Both of these layers have their quirks.  In the temperature screenshot above, you can see that the South side of Giraud (unshaded area at the bottom of the image) appears inexplicably colder than the North side.  In the precipitation screenshot, the storm appears to be very respectful of the New Mexico state line, presumably because each side is coming from a different forecast office.  In these situations, it's sometimes best to go straight to the source.

The "NWS Forecast Grid" layer is an interactive data layer showing the center points of the NWS forecast grid squares.  At zooms 12 and above, there is a dot for each grid square; at 11 and below, only a portion of the squares are shown.  Mousing over a grid point shows the elevation I'm using for temperature interpolation, which is just a guess based on averaging points in the grid and will precisely match the elevation the NWS uses.  Clicking on a point opens a new tab with the NWS hourly weather chart:


Going back to the Giraud example, let's take a look at the source forecast data:


Remember that the temperature is representative of the entire 2.5km grid square and not that exact point, but the North side is forecast to be warmer, at higher elevations, than the South side.  Mousing over the 47 degree point shows that I'm estimating it as 11375'; clicking on it, the NWS shows 11132'.  I estimate the 37 degree point South of that as being at 10325', and NWS says 10696.  So it's not just a quirk in my interpolation - they are forecasting the North side to be significantly warmer, despite being higher.

Is that an accurate reflection of reality?  I couldn't say, but I would guess the NWS forecast lacks that level of pinpoint precision.  But I do know that I'd rather have all the data from which to make my own informed decision, than a somewhat arbitrarily chosen point forecast or two.

Wednesday, July 19, 2017

Announcing Team Accounts

Team accounts are a new feature designed to both improve data sharing within an organization, and allow organizations to buy a single subscription covering all their members.

Maps can be saved to either an individual's account or the team account, and both sets of data are rolled together in a user's account dialog.  Because the "your maps" table is sorted chronologically, new work saved to the team account will automatically bubble up and be visible to other team members:


Icons and custom map layers saved to the team account will automatically be visible to all team members as well; if you regularly use specialized icons or custom layers, this is also a great way to push them out to team members - no more emailing around GPX files with the latest copy of your team's customizations.

Team accounts come with a dual pricing scheme - $500/yr for organizations consisting primarily of unpaid volunteers, and $2000/yr for organizations primarily made up of paid professionals.  Both versions cover organizations up to 100 people; for larger organizations, please email info@caltopo.com.  If there is interest, I may be able to add a cheaper small-business option for companies with only a handful of employees.

Beyond data sharing and customization, this price includes pro-level features for all team members, and a site license for offline use covering both team computers and personal computers owned by team members.

To sign up for a team account, go to https://caltopo.com/group/join or https://sartopo.com/group/join.  For more information see the knowledge base article on team accounts. 

Individual accounts remain unchanged.

Wednesday, July 12, 2017

SAR Grab Bag

Once again, although the blog has not been updated, the wheels of progress are still turning at CalTopo HQ.  Here's a quick run down on some new features that, while available to everyone, will mostly appeal to SAR users.

Custom Icons
CalTopo has long supported custom icons by allowing you to enter a URL on the icon chooser, but the built-in limitations made this clumsy - you had to host the image elsewhere, and re-enter the URL for each new icon.  Now, you can upload the files directly to CalTopo and tie them to your account (limited to pro-level users only):


Once uploaded and configured, your icons will appear as a new row in the icon chooser:


Variable Opacity
Lines and polygons now feature an opacity setting, so that you can create partially-transparent features that allow the underlying basemap to show through.  On the SAR side, opacity lives on operational periods and cascades down to child assignments and tracks, making it easy to show existing search efforts without completely obscuring the map.


Buffers
There's a new "buffer" object type that lets you trace along a linear feature and create a polygon encompassing all points within a specified distance, making it easy to draw an assignment that actually covers everything within 200' of a trail or drainage.


After selecting New -> Buffer, you'll see a dialog asking you to select the buffer's size:


From there you'll see the normal line drawing dialog and UI.  When you finish drawing, the line gets converted to a polygon.  At the moment, this is a one-way operation: as with sectors, once a buffer is created it's just another polygon.  There's no way to alter the line that the buffer was created from.


Snap To Existing Lines and Polygons
Playing well with buffers, the auto-draw tool has been extended to let you snap to existing map objects like lines and polygons.  For maps built on non-overlapping gridded areas, like a search planning map, this makes for a much cleaner end product than trying to manually trace out two polygon boundaries to match each other.



Assignment Letters
SAR planning is still a wild west of differing approaches, but it's fairly common for teams to want to identify segments via letters, and teams assigned to those segments via numbers.  There are a number of ways SARTopo could handle this, including having separate segment and assignment objects, but I decided to go with the simple and universal approach of giving assignments two properties: a number and a letter.  You can use both, or either, and the map labels should reflect this intelligently.  The 104 form has also been updated with a space for the letter.

Assignment Summary
A feature I'm particularly excited about is the assignment summary form, accessible via the assignment bulk ops menu.

For background, common practice on searches is to prepare a separate assignment form for each team, which is also used to capture team members' names.  This is generally done using multipart carbonless copy paper; the team gets the top (most legible) copy, and the CP keeps the rest.  Capturing the names of each person on an assignment is important for safety and accountability reasons, but it's hard to print onto loose-leaf carbonless paper.  Doable, but easy to mess up - so the forms are often filled out by hand, even if some of the information is already in SARTopo.

From my involvement with large multi-day incidents, getting large stacks of assignment forms properly filled out with the right details is one of the major tasks keeping people up late and depriving them of adequate sleep.  And since teams in the field only have their own form, it limits their situational awareness.

It's also common practice to include an assignment-specific map with with each form, a feature SARTopo supports.  However, for both logistical and situational awareness reasons, I've been pushing people away from that feature and towards printing a tiled multi-page incident map to be included in the IAP - which is the way wildland fire incidents have done it for some time.

The assignment summary form extends this approach to SAR 104 forms - an approach that's also in line with the way the rest of the ICS world works, albeit in a more paper-friendly fashion.  Pertinent information for all the assignments is condensed into a page or two, suitable for inclusion in an IAP without burning through too much paper.  Carbonless forms would still be used for tracking team members' names.


As with the "print a coordinate list" feature, fields to be displayed are user-selectable.  You can print full details and a mini-map per assignment, or condense each one down to a single row.

End result:


Along with the IAP summary, there's a separate deployment summary that makes use of newly-added team size and priority fields to create a nice single-page reference for operations to use while forming and deploying teams: