Tuesday, November 20, 2018

New MapBuilder Roads Option

One feature request I get fairly often is a better printable roads layer, primarily from SARTopo users for use in both urban searches and disaster response.  The USGS layer is out of date, MapBuilder doesn't show enough street names, and OpenStreetMap gets shrunk to the point of illegibility when printing.  I've always assumed that the proper answer was to bring the default OpenStreetMap style into the MapBuilder rendering stack, which would allow scaling it to match a PDF's resolution - but because that introduced a number of complications, I kept kicking the can down the road.

Necessity is the mother of invention, and a little over a week ago I was called to the Camp Fire in my role as a SAR team member and found myself needing to generate a lot of street maps.  One requirement was that they display as many street names as possible, so that responders from around the state could use them for navigation despite not being familiar with the area or having a cell signal.  A clean white background was also important to allow marking up maps in the field.  Fortunately I'd just wrapped up a MapBuilder import and had a second rendering server sitting idle with the new database, so I was able to hack together a new layer style quicker than expected.

There are a lot of lessons learned from trying to apply my wilderness SAR background to an urban disaster, but one of those is that the new roads layer is here to stay, and I'm sorry I didn't have one out sooner.  It will probably get tweaked further and eventually added as a primary layer, but for now it's only available as a custom MapBuilder layer (pro level account or higher required):

Check "compress text".  Recommend setting roads to "thin lines".
With the "compress text" box checked, a number of changes are made to support showing as many street names as possible:

  • Suffixes are dropped (drive, road, street, etc)
  • Font size for road labels is reduced
  • Labels are allowed to bend more
  • Labels are spaced closer together
The Oakland hills with most streets labeled at a1:12k scale
The result isn't exactly pretty compared to something like Google Maps, but it's fairly effective as a first response tool since it will show a lot of street names at usable map scales like 12-24k.

San Francisco at 1:12k

Even at 1:24k, a lot of Berkeley street names still get shown

The layer didn't exist 10 days ago, and hasn't seen a lot of careful development or evaluation, so feedback and suggestions are definitely welcome.

Tuesday, September 18, 2018

Major Offline Improvements

A new version of CalTopo Offline and SARTopo offline has just been released (4151), with several major improvements:

1.  Real-time syncing between offline and online versions.  All of your maps from the website will show up in the offline version, and changes will automatically sync between the two.  If you make a change while offline, it will be queued until you regain internet access.  This is helpful both when starting a project online and then taking it into the field, and also when working in an area with unreliable internet, so that you can keep changes synced to the outside wold without being at risk if the internet dies.

2.  New map downloader that downloads offline map data within Cal/SARTopo Offline.  This makes it much quicker and easier to download large areas than the old web-based downloader.  Because the new downloader offers multiple resolutions and is built on a 15x15 minute grid, it also gives you more control over how large your offline maps are.

3.  Revised layer selection.  Elevation, slope and aspect data are all rolled into a single layer.  Shaded relief and slope angle shading are automatically computed from elevation data, and no longer need to be downloaded separately.  Most excitingly, MapBuilder Topo, Hybrid and Overlay layers now have full coverage for the contiguous US.

4.  Layer caching.  Map areas viewed in the offline version when you do have internet access are cached for later use offline.  So even if you haven't pre-downloaded an area, it may still be available when offline.

Tuesday, September 11, 2018

Come Work For CalTopo

CalTopo is growing!  I'm looking for a contract graphic artist and a full-time UI developer.  Come work on a product you actually use, in a job with significant flexibility.  For more information, email mjacobs@caltopo.com.

This could be your office (deck not included in job offer, applicant to supply their own)

Graphic Artist

I'm looking for a graphic artist to take on a contract project revising and expanding the current icon symbol set.  If things go well there's potential followup work (UI icons, MapBuilder symbology, general UI design), but nowhere near enough to add up to a regular job.

I'm also looking for assistance creating a logo, whether that's the same designer described above or a specialized professional.

UI Developer

The baseline for this full-time remote position is a competent developer with a solid web UI skillset, able to be turned loose on projects with minimal supervision.  Preferably, you can also take on UI and interaction design, coordinating with a graphic designer on the visual pieces as necessary.  Ideally, you can tackle the entire UI development cycle from graphic design through development.

The following are not required but are considered pluses:

  • A CalTopo user or outdoor enthusiast.  This is a position where domain knowledge matters.
  • California or Washington residency.  As a full time remote employee, CalTopo LLC will have a business presence in your home state.  Some states are easier to work with than others, but CalTopo is already registered with CA and WA.
  • OK with occasional travel to Truckee, CA (potentially up to several days a month, generally less).  Although the position is remote, face time can be important for collaboration, particularly at the beginning of a design.
  • First response experience, particularly in the backcountry (SAR, fire, ranger, etc).  This is an important and growing customer base for CalTopo, and as with the recreational side, domain knowledge is useful.
  • Mobile development experience. 

Friday, October 13, 2017

Wind Forecast Layer

When I added the weather forecast layer this summer, I mentioned my frustrations with the way the NWS requires you to bring up endless point forecasts to get an accurate picture of local precipitation amounts.  With the Northern California fires, I'm running into the same issue with wind - I'll read about a forecasted red flag warning, but the Napa and Santa Rosa forecasts barely hit double digits.  I'll admit that I had no idea wind speeds were so locally variant - or at least, no idea that the NWS forecast grid captured such small-scale variations.

24hr wind gust plot.  From 15mph in St Helena to 50mph on Mount Hood just 3 miles away.

Fortunately I already had wind speed mapping on the back burner - it didn't get deployed along with the earlier temperature and precipitation work, but most of that code was reusable.  I can't show wind direction - even if I were to render directional arrows on the map, frequent direction changes make it impossible to show a single meaningful 24hr or 36hr wind direction.  However peak forecasted wind speeds and gusts are shown for 1hr, 6hr, 12hr, 24hr and 36hr intervals, using the same green-yellow-orange-red-purple-blue-black gradient scale as the temperature layer.

One option that does provide wind direction is the crowd favorite windy.com.  However, windy does not reflect the small-scale variations in the NWS forecast grid (see below).  As with temperature and precipitation, it's an open question as to how accurate the NWS grid variations are, but I like to provide as much raw data as possible and let users draw their own conclusions.

Windy.com plot of the same location
The wind layer is a checkbox option alongside temp and precip:

The forecast grid option immediately below will also show point speeds (remember, the point is simply the center of a grid square, and the forecast applies to the entire square), and clicking on a point will bring up the hourly weather chart.

I feel like I've pretty much run through the backlog of fire-related items I had sitting around, so I think this will be the last major layer change in response to the Northern California fires.

Update: there are now two wind layers, the "max wind speed" layer as described above, and a "wind plot" that provides forecasted directions and speeds for specific points in times, at 3 hour intervals to 12 hours, and then 6 hour intervals to 36 hours.  The same color chart is used for speed, with short lines tracing direction.  The length of the lines has no meaning.

High Resolution Post-Fire Imagery

Thanks to a generous offer from a contact on the Google Crisis Response team, CalTopo is temporarily displaying high-resolution imagery for portions of the fire-affected areas in Napa and Sonoma counties.  This imagery was taken by DigitalGlobe on Wed the 11th, and although the fires are still ongoing, it provides some degree of insight into what had burned by that point.  While there is a lot of smoke present, in many areas it's clear enough for a house-by-house assessment.

Because the original imagery is false-color infrared (which gives vegetation a strong red tint), I'm converting it to black-and-white to avoid any possible confusion between the red tint and currently burning areas.  I'm told that true-color imagery will be available at some point on Fri, and hopefully I'll be able to upgrade the layer to match.

Available imagery footprints, with fire outlines shown in red.

Step 1: Go to https://caltopo.com/l/16PD

Step 2: Enter your address in the search field at the top of the page and click GO:

Step 3: Use the zoom control at the top left of the map to zoom in:

Step 4: Mouse over the layer selector at the top right (it should read "Hybrid +1"), and in the window that comes up, adjust the "NorCal Fire" opacity slider to expose the pre-fire imagery:

It may also be helpful to temporarily turn on the "Current Fire Activity" checkbox in order to see the fire perimeters:

I've put some temporary infrastructure in place to help handle this project, but if the site is impacted too much, I may need to make some changes.  The shortlink at the top of this article (https://caltopo.com/l/16PD) will always point to the right place - if you share a URL, please share that.

Wednesday, October 11, 2017

NorCal Fire Response

The quick summary:

  • If you are seeking information on the Northern California wildfires, or trying to understand CalTopo's current fire activity layer, please see this blog post.
  • Edit: big thanks to the Google crisis response team for rapidly lifting the map quota.  The OpenLayers warning no longer applies.  CalTopo will be intermittently running on OpenLayers instead of Google Maps.  If the zoom controls at the top left of the map viewer are blue instead of white, Google layers (map, terrain, satellite) will be unavailable, and text based location search will not work.  You can use the search bar to locate lat/long or UTM coordinates, but not named places.
  • The current fire activity layer now includes VIIRS 375m detections as well as MODIS.
  • CalTopo now sports a new mobile UI.  If you have issues with it, please let me know.
The details: in case you've been living under a rock since Sunday night, California is on fire:

Sunday night's VIIRS capture: fires that look like cities

On Tue morning, CalTopo started seeing increased visits from people looking at the fire activity layer, and pageviews rapidly grew to well above normal.  This caused some scaling issues, but barring another order of magnitude in growth, those are temporarily solved with a combination of software fixes and throwing hardware at the problem.

The larger issue is that I ran into the 100k pageview limit for the Google Maps standard plan, and CalTopo stopped loading.  I've fixed this by switching to OpenLayers, which is the map viewer used in CalTopo Offline, but it's not a perfect substitute: in addition to missing Google's map layers, text-based location searches (like "santa rosa, ca") will no longer work.  The version of OpenLayers I'm using is also old and heavily customized, so there may be some compatibility issues with touch gestures on mobile devices.

I'm working with Google on fixing the issue, but in the meantime, I'll turn on Google Maps at some point in the morning, and leave it on until the quota runs out.  If the zoom +/- buttons at the top left of your screen are white, you're using the Google Maps viewer and everything should function normally.  If they're blue, it means you're using OpenLayers.

In order to better serve people looking for fire information, I've also held a marathon coding session to wrap up and deploy two back-burner upgrades.  The first is the addition of VIIRS 375m fire detections alongside MODIS.  At more than double MODIS's resolution, VIIRS should provide a more accurate picture of the fires' behavior.  Although VIIRS suffers from a limited number of passes (just like MODIS), adding a second source will increase the layer's all-important update frequency.  And certainly the most common question I've gotten today is "when will the layer update"?

Looks just like MODIS, but with more points

The current fire activity layer was already customizable (smoke vs no smoke); now you can choose all sources (a mix of MODIS and VIIRS on the same map) or view them one at a time for a cleaner image.

The second change is a revamped mobile UI.  Normally I'd hang on to a change like this for another month or two in order to test it thoroughly and smooth out the edges.  However, the vast majority - > 90% - of people seeking fire information are doing so on their phones.  And, the previous mobile UI was non-functional enough that I'm unlikely to make things worse.  I know it's 2017 already, but now the CalTopo mobile site supports 90% of the features the desktop site does.  More details in a later post once I get things cleaned up, but in the meantime let me know of any bugs.

Tuesday, September 5, 2017

CalTopo Guide to Wildland Fire Information

NOTE: I wrote the bulk of this post in July, but shelved it due to some pending layer changes.  With the western US currently on fire, it seemed like a good time to finally wrap it up, although some of the references are no longer timely.

Here in Truckee, smoke from the Detwiler fire - 100 miles away - drifted in two nights ago.  A good reminder that I've been meaning to do a post on wildland fire information, using both CalTopo and other tools.  Whether there's a fire burning near your house or, like me, you're just trying to dodge smoke in order to exercise, getting accurate information can be a frustrating experience if you don't know where to look.

Not to pick too heavily on the mainstream media, but if I were trying to learn about the fire currently blowing smoke over the Sierra, the top Google News result is a recently published LA Times story:

The headline is a bit sensational - the exact quote in the article is that weather satellites spotted "explosive fire behavior" - but that's the same as it ever was.  Despite the article's many small but not terribly useful details, such as firefighters tackled 2- to 4-foot flames, the core information is wanting.  The fire perimeter map is over a day old, and their included GOES satellite image is indecipherable to anyone who doesn't know what they're looking at, myself included:

So, where I do go for information?

First stop is a set of "real time" (technically, several per day) CalTopo layers sourced from the two MODIS satellites (named Aqua and Terra) and VIIRS.  Current visible-spectrum imagery from the satellites is available through the standard layer dropdown, in both standard and relief-shaded versions.

Generally one of the satellite passes has better resolution and color quality than the others, and the current day's imagery is often not available until late in the afternoon.  Here's yesterday's Terra pass over the Detwiler fire:

As a side note, even smaller fires and controlled burns often produce smoke that's visible to the satellites.  Something to keep in mind the next time the news excitedly proclaims that a fire is now VISIBLE FROM SPACE, along with a MODIS capture like the one above.

The MODIS satellites also capture infrared imagery, which is processed and used for hot spot detection.  Generally hot spot data comes from a series of midday and overnight passes, capturing fire behavior at two points in the diurnal cycle.  These hotspot detections form the core of CalTopo's current fire activity layer:

Each dot represents a hotspot detection.  Color indicates age - red is detections less than 12 hours old, orange is 12-24 hours old, and yellow is 24-48 hour sold.

MODIS data is stitched together from a number of satellite passes, with each pass covering a long and narrow swath of the planet.  The image resolution is nominally around 1 kilometer per pixel (ie each pixel in the MODIS image represents 1 square kilometer), however the resolution drops as you move towards the edge of the pass.   As a result, some hotspots have a larger error radius than others.  Zooming in on the map shows that the error radius of each hotspot as a semi-transparent circle.

Technically, the pixels are in a rectangular grid, and each hotspot detection comes with separate X and Y error values. Due to the complexities of properly mapping this grid, I average the X and Y values into a single, circular error radius:

In the above image, the older (orange) points have a much larger error radius than the newer (red) ones.  All we can know is that there is a hotspot somewhere inside each circle; the entirety of the orange circles in particular is probably not on fire.  It's also worth noting that because the MODIS satellite is generally taking a semi oblique image, rather than looking straight down, terrain with heavy vertical relief can throw the estimated locations off - a single detection on the far side of a steep river canyon may be an erroneous interpolation and not actually indicate that the fire has crossed over.

Zooming in more shows not only the time of the detection, but the temperature, radiated power, and confidence that the detection is actually a fire.  While it can be fun to geek out on, I'm certainly not qualified to make any useful predictions based on that information.  If anyone reading this is, please chime in.

Infrared hotspot data is also available from VIIRS, and I'm looking at adding that along with other data sources such as GOES, so the accuracy of the fire activity layer may improve for next season.  Both MODIS and VIIRS hotspot detection data is available straight from the source at https://earthdata.nasa.gov/earth-observation-data/near-real-time/firms/active-fire-data.

The current fire activity layer also includes smoke information from the NOAA hazard mapping system (HMS), typically updated every several hours.

MODIS is a good rough indication of a fire's activity, but it's not perfect.  In the US, a fire's incident management team keeps track of the actual fire perimeter using a combination of aircraft observations, including nighttime IR flights, and GPS tracks recorded by teams on the ground.  These fire perimeters are typically uploaded once a day to GeoMAC, and then pulled into the CalTopo fire activity layer as crosshatched yellow polygons:

Detwiler perimeter, a little hard to see through the smoke data also included in the fire activity layer.
CalTopo also sources historic fire perimeters from GeoMAC for the fire history layer.  Wildland firefighters might scoff at the notion of calling 15 years of perimeter data "historical", since they use decades' worth of data to analyze fire behavior in a given location, but I have yet to find a better nationwide source.

Although not guaranteed, the fuel load in recently burned areas will generally be lower than in areas that haven't burned for a while.  Old fires may also be bounded by existing fuel breaks that can help stop the spread of a new fire.  Because of these factors, it can be useful to look at existing fire perimeters when trying to figure out how and where a fire might spread:

This part of the Shasta-Trinity has burned a bit
As a reminder, subscribers can pull these layers (along with all of CalTopo's other layers) into Google Earth via super ovlerays for 3-D viewing:

If you want to step outside the CalTopo bubble, one of the first questions to ask is "who is responsible for fighting the fire?"  Once a fire gets large enough this is generally available from the news, but it's also helpful to look at the CalTopo land management layer to figure out whether a fire is on federal or non-federal land.  This is a good first-order approximation, but keep in mind that fires on non-federal land could be the responsibility of either a state or local agency.  Agencies will also swap responsibility for some areas depending on who has nearby resources, so responsibility areas do not map 1:1 with land ownership.

Detwiler fire approaching, but not yet on, Forest Service land
For fires on federal land, my first stop is generally InciWeb, which has all kinds of good information:

InciWeb home page / incident list

Each incident page has a number of easy to miss tabs near the top.  The news articles (aka press releases) contain dry but substantive and useful information, and the maps tab contains links to the actual incident maps.  Generally these are updated daily as part of the incident action plan (IAP), and not only show the fire perimeter, but break it down into established fire line and uncontrolled fire edge.  Coming from a SAR background where incident data is often restricted due to law enforcement concerns, the ready availability of fire information is refreshing:

Southern edge of the Whittier fire 

If InciWeb  doesn't have the fire listed, or if you want to dig really deep into the details, my next stop is the national interagency fire center FTP site at http://ftp.nifc.gov/incident_specific_data/.  It takes a little digging (based on both location and jurisdiction) to find a specific fire, but the NIFC server generally has more detail than inciweb, and at least in California, details for state-managed fires.