Friday, December 12, 2014

Feature Week: Print to KMZ

I've received a few emails asking where the KMZ link in the left bar went.  It's been replaced by a print-to-KMZ option:

Why?  Because instead of just map layers, KMZ exports can now have markers and lines "printed" on them.  These are not traditional KML placemarks - for that, use the standard Export option.  Instead, they're rendered as graphics along with the map.  This can look a little fuzzy in Google Earth:

However, it really shines when used as a Garmin custom map.  You can draw far more lines on the map than most GPSs can handle as tracks or routes, and it allows you to create more colorfully marked up GPS maps.  Example screenshot below:

Thursday, December 11, 2014

Feature Week: Bulk Operations

If you've ever wanted to change the color on a number of lines at once, the new bulk operations feature will nicely complement folders.  The Markers and Shapes boxes, along with folders, now have a Bulk Ops link at the bottom.

Bulk operations can only be run on one object type at a time, so if you click Bulk Ops on a folder with both Markers and Shapes, you'll have to choose.

Once that's done, you'll see a full page dialog listing each object with some options at the bottom.  Click or shift-click to select a group of objects.

The Print Individual Maps option is only available for saved maps.  It will open a multipage PDF in a new browser window, with one page per object.  Each page is automatically centered on the object and scaled to fit, with a minimum scale as specified.  The default is 1:12k, which strikes a nice balance between capturing small map details and not losing context.

Change Attributes will bring up a second full page dialog, similar to the object's details dialog.  All fields start off blank, any changes made will be propagated to the selected objects.  Beyond setting marker and line styles, you can move objects between folders en masse.

Delete Objects will obviously delete the selected objects, although not without bringing up a confirmation window.

Wednesday, December 10, 2014

Feature Week: Folders and WYSIWYG Printing

Over the years I've had a number of users ask for a way to organize markers, lines and polygons within a map.  There have also been requests for a way to print only some objects, or to print markers with labels and shapes without, and so on.  Today's post covers a new way to do just that: folders.

Start by adding a folder to the map the same way you would any other object: right-click on the map background or click + Add New Object in the left bar.

The details dialog for markers, lines and polygons now has a folder dropdown.  Choose a folder at creation time, or move objects between folders the same way you would set any other property.

Individual folders will show up as boxes in the left bar alongside other object types.  Objects without folders will remain in their object type box, e.g. Markers.

As with the Markers and Shapes boxes, you can use the checkbox to show or hide all objects within a folder.  If you want to hide labels for a folder, perhaps because they're too noisy, the settings box now has a third, custom label option.

When set to "some", each folder gets a label toggle at the bottom.  Click on it and "Labels Shown" will change to "Labels Hidden".

Alongside these changes, printing has moved to a what-you-see-is-what-you-get (WYSIWYG) model: what you see on the screen is what you get when you print a map.   If an object is visible when you click print, it will show up on the print.  If it's hidden, it won't.  If it has a visible label, that label will show up on the print.  If the label is hidden, it won't.

Because this requires a form POST, I can now only offer the "share or save this configuration" link for a blank map, not one that has objects on it.  However I think the extra flexibility when printing is worth the cost.

Monday, December 8, 2014

Feature Week: Automatic Syncing and Collaboration

The past several weeks have seen significant behind-the-scenes performance upgrades to CalTopo, including mass migration of lines and polygons to a more efficient storage format.  Some of these changes were not without risk, so if anything looks off, please don't hesitate to let me know.

Only time will tell, but it looks like I'm out of the woods on performance work, and that has allowed me to roll out some long-deferred changes.  To celebrate, I'm going to run a series of posts highlighting new features.  The first one is automatic syncing and collaboration.

Without running afoul of trademark law, I've often tried to describe CalTopo as being like "Google Docs for maps".  While not as fully featured as a desktop program, from the beginning it enabled collaborative sharing and editing of maps in a way that simply wasn't possible with software like Nat Geo's now-discontinued Topo!.  However, for performance reasons, true Google Docs style "everyone sees all changes instantly" editing remained locked behind an optional checkbox in the sharing dialog.

No longer.  Starting now, saved CalTopo maps will automatically pick up on changes as they are made by other users.  Unlike Google Docs, there will still be a bit of a delay - from 5 seconds for actively edited maps up to 30 seconds for maps that have not been changed in the last 10 minutes.  After a couple weeks I may be able to dial this down depending on server load.

So how do you start this shared editing?  Time for  a quick refresher.

  1. The map needs to be saved (URL ends with /map?id=ABCD).  If you're working on an unsaved map, click the red Save link at the top of the left bar.
  2. Click Share.
  3. In the permissions dialog that comes up, either give write access to all users who know the map's URL, or set a password and give write access to users who know the password.
  4. Send the map's URL (e.g. to a friend.
  5. If you set a password, they will need to click on the Share link and enter it.

That's it.  Happy collaborative editing.

Friday, November 7, 2014

Mo Users, Mo Problems

Over the past several weeks, I've been getting progressively more frequent emails from the Binary Canary monitoring service letting me know that CalTopo was down.  It was sometimes unresponsive for extended periods of time, but since it generally came back on its own, I did what any good developer would do and ignored the problem until it reached red alert levels.
After spending 2 days a very long ways away from cell reception for SAR, I came back to a flurry of emails, including another Binary Canary outage warning - and CalTopo still appeared to be down, 5 hours later.  This was the tipping point; something had to be done.

Throwing together some changes and deploying them with a half-assed amount of testing was something, so that's what I did.  While it solved the performance issues, it broke imports and exports - something which CalTopo users let me know in short order.  Now that the fire drills are over, here's a quick recap of the issues.

One problem was in the way I sent line/polygon points from the server to the browser.  Instead of using a numerical array, each line point was a separate object.  This isn't a big deal in and of itself, but the JSON library I'm using to serialize everything created a heavyweight map object for each lightweight waypoint object, resulting in a huge amount of object churning relative to the memory available.  Points are now sent as an array, which doesn't have that overhead.

The second problem was - and possibly still is - with supplying cursor elevation.  Because I push elevation data to the client as a plain-text array of numbers, the 256px tiles are significantly larger than PNG or JPG map tiles.  As an added bonus, because elevation data went through the same server-side process as shapes and markers, the browser was instructed not to cache any of it.  Every page load generated a flurry of elevation requests.

One small checkbox, one giant bill.
Since turning the "Include Elevation" checkbox on by default, it turns out I was spending $150/month just to send elevation data to clients.  Oops!  I have a rather hefty Amazon bill, most of which is reimbursed by smartphone apps that license my maps.  Amazon makes it difficult to separate EC2 data charges (my server) from S3 data charges (map tiles), which is why a number several times larger than my monthly beer budget had slipped by unnoticed.

Now that elevation data is getting cached, I'll give it a week or two and decide whether I need to go back to having the checkbox disabled by default; the answer will probably be yes.

What this incident has made clear is that I need to find a way to make CalTopo commercially viable.  I've asked people to send donations to BAMRU because I thought that as a 501(c)3, it would encourage more liberal giving.  However, even if the donations BAMRU receives were funneled directly into operating the site, they would still be insufficient to cover my monthly operating costs.  Fortunately there are commercial map licensees for that, but while they pay me enough to cover my fixed costs, they're not really not a profit center.

When asked about my plans for commercialization, I've always maintained that while I don't want to cover the site with ads or turn it into useless baitware, I'd rather offer a better experience at a reasonable price than a mediocre one for free.  If it allows me to devote more time to the site and provide a better feature set, some kind of paid model would be an all around win.

After a lot of delay, I think it's time to go down that road.  More details will be provided as I figure them out, but I think the following is a safe high-level picture:
  • Most of the site will always be free.  From a purely self-interested perspective, I want to be able to hop on someone's laptop and sketch out a map without having to log in, verify my account, etc.
  • Any paid features would be part of a subscription rather than pay-per-use.  A number of people have suggested pay-per-print, but that seems rather petty; I'd prefer that people not have to ration their usage.
  • I'll try to align paid features with my cost structure, i.e. the features that cost me more time or server resources to support will be the ones I charge for.
  • Everything, including offline map data, will continue to be free for SAR and other first responders.
  • I'll be selling related items like GPS maps and 3D posters as an additional funding source.
So with that in mind, what are some things that might become part of a subscription?  Nothing is settled, but ideas that come to mind include large PDF sizes like 24x36, PDF links that last longer before expiring, WMS map access and having elevation data show up by default.  I'd also like to integrate non-free map sources (i.e. I need to pay the map provider), and a subscription seems like the logical way to tackle that.

The goal is to devote more energy to CalTopo, not to let it wither while I try to wring money from an ever-shrinking user base - though that seems to be a popular approach, too.  Paid features are an experiment, and if they turn out to be a bad idea then I'll drop them.  Changes will be neither drastic or immediate, and you don't need to worry that PDF generation or some other critical feature will suddenly wind up behind a paywall.

All of that said, I'm open to suggestions.

Monday, October 6, 2014

Daily Dose of MODIS

I've written previously about the MODIS satellites and their constant imaging of the planet; CalTopo's fire activity layer is derived from their infrared sensors, with a lot of post-processing by the federal government.  There's now a new pair of layers that make use of the visible-spectrum sensors: current and archived MODIS imagery.

The Latest MODIS layer can be found under the Aerial Imagery group.  Imagery is acquired twice a day, but note that the first pass may not happen until the afternoon.

All quiet in California, but a little foggy on the coast.

The layer has a nominal resolution of 250 meters / pixel, but since the imagery is often not tack sharp, usable resolution will be less:

Not quite enough to read license plates.

The latest image may be covered with clouds, or the area you're interested in may be especially blurry if it was at the edge of an imaging strip.  That's where the archived imagery comes in, as a new configurable layer option:

Choose which pass you want (the satellites are named Aqua and Terra), the date you want imagery from - today, yesterday, or any day back to May 2012 - and you're off and running.

You may be thinking "so this is a fun toy, but how does such a low resolution image help with backcountry trip planning?"  To answer that question, lets dial the clock back a couple weeks to September 16, when the King Fire was going strong:

CalTopo's fire activity layer pulls in smoke polygons from a separate source, but there's nothing like seeing it with your own eyes.

As another exercise, a light storm blew through the Sierra shortly before some friends headed to Desolation Wilderness last fall.  They asked me how much snow to expect, and I erroneously under guessed as Truckee had already melted out several days prior.  If only they'd had easy access to MODIS imagery, it would have been obvious that their destination was still coated in a thin layer of the white stuff.  As an added bonus, you can make out smoke plumes from controlled burns along the western slope of the Sierra:

This has barely gone live and I'm already in the habit of checking it once a day, just to see if anything interesting is going on.

Monday, September 8, 2014

End to End: Closed Contour Yosemite Maps

A few years ago, I ran across Dan Cervelli's excellent Closed Contour map of the Sierra.  At the time I was just getting started with CalTopo and learning my way around GIS, and Closed Contour really opened my eyes to the cool stuff you can do with Mapnik.  I had no idea it was so simple to take some shapefile data, add a few rendering commands, and then boom, beautiful looking maps.  At least, the tools are that simple - obviously there's still a lot of talent and hard work involved in producing the final product.

The original Closed Contour map was built on a projection and tiling system incompatible with Google Maps, so integration into the universe of sites and apps using that as a de-facto standard was not possible.  Dan has a new Yosemite map built around this standard, and I thought it would make a great end-to-end demo of some things CalTopo can do with "web mercator" tile sets.

Step 1: Add a Custom Layer

The more maps, the better - CalTopo is built around the idea of combining different map layers into a single coherent picture.  Unfortunately, I have to balance supplying layers of regional or special interest against providing a reasonably compact, easy to navigate UI.  The middle ground I've settled on is to make it easy to add custom layers from a tile URL or WMS address.  You can also choose from a preset list of useful layers, which Closed Contour has now joined:

Closed Contour now appears in the layer dropdown alongside the standard CalTopo layers

and provides a snazzy backdrop for some route planning, complete with trail mileages

You can also skip these steps and just visit to take a look.

Step 2: Make a PDF

I've already covered printing multipage PDF map packs and loading Geospatial PDFs onto your smartphone, so lets just jump ahead to the results:

The Half Dome hike fits onto a single 1:24k scale 8.5x11 print.  Compare Closed Contour, above, to the USGS 7.5' topo below:

I can also load this into Avenza PDF Maps to plot my location and record tracks when out in the field:

Step 3: GPS and Google Earth

Both Google Earth and newer Garmin GPSs know how to read layers exported from CalTopo as KMZ files.  So lets go make one:

This gives me a better GPS map of Yosemite than money can buy:


And some pretty nice 3-D visualizations:

Friday, August 15, 2014

PDFs get Geospatial

Printing Geospatial PDFs that can be used in programs like Avenza PDF Maps is an oft-requested feature that's been hanging over my head for some time.  Having looked into it and abandoned the effort, I've lived life dreading the prospect of wading back into the fray.  However I took another crack at it this week and, armed with a better understanding of the PDF spec, it turned out not to be so bad.

Starting now, all single-page PDFs generated by CalTopo are Geospatial PDFs.  Edit: Looks like I jumped the gun a bit.  There were some compatibility issues with the iOS version of Avenza that have now been fixed.

Complementing this is a new short-URL feature that makes it easier to share CalTopo PDFs or load them onto a phone.  Step 1 is, as usual, to click Print and then Create a PDF:

Underneath the Generate PDF button is a new Generate Link button.  This behaves just like Generate PDF, but instead a new browser tab opening with a long query string or a blank form post, it opens with a short URL.

Anyone with this URL can now download your PDF; no need to save it to a computer and then share using a 3rd part hosting service.  While the print page indicates that links expire in 30 days, I may keep them around longer if they don't occupy too much space - currently I have to go in and purge them manually.

The only reason for a separate Generate Link button is to avoid cluttering my database with PDFs that will never be shared.  If that proves to be a non-issue, I'll merge them into one button that always generates a short URL.

I have no affiliation with Avenza, but their free PDF Maps app allows you to import directly from the CalTopo URL:

Thursday, June 19, 2014

Revised Marker Icons

In keeping with the recent line style update, I've made some changes to marker icons.  While a few icons have been added, the focus was adding colorability and rotatability, and laying the groundwork to add additional icons at a later date.

Instead of showing all possible icons, the marker dialog now shows the icon's style, and optionally its color and orientation.  Click on the style swatch to bring up a full-screen dialog similar to the line style dialog:

Mostly this is a retread of the previous set of icons.  A few wildland fire icons have been added, along with a letter-in-circle option.  However unlike the previous all-in-one dialog, this one has room to grow.

The color and orientation sections are selectively enabled for icon styles that support them.  Currently color is supported for all the circle and arrow icons, and rotation is supported for arrows and the wildland fire line breaks.  Click on the color swatch to bring up a color picker, just like with lines and polygons.

For icon styles that support rotation, you can set the orientation by entering a number of degrees, or by clicking on the axes overlaid with a directional arrow.

Thursday, May 29, 2014

USGS Stream and Reservoir Gauges

Building on the code already in place for Snotel, I'm pleased to announce another real-time datasource: USGS water gauges.  While it's more of a frontcountry source than a backcountry one, streamflow information can still give you a good picture of the snowmelt situation and help determine whether rivers are easily crossable.  Of course boaters use it all the time, but large rivers only make up a small portion of the network, and those gauges are better covered by sites like

that's a lot of gauges

There are 7500 stream and reservoir gauges, v. 850 snotel sites.  I haven't modified the code to be more selective about how many stations get displayed at once, so users with slower, older machines may find their browser slowing down after enabling the water gauge layer.  If this proves to be problematic, let me know - if enough people complain, I'll look into fixing it.  On my 3 year old laptop, it was a non-issue.

Much like the snotel layer, zooming in displays the current real-time value for individual gauges.  Streams show the flow in cfs (cubic feet per second), and also the water temperature in degrees F, if available.  Reservoirs show the current capacity in acft (acre feet).  When you mouse over a station, the tooltip shows the time at which it was last updated.

Click on a station to bring up a graph of the last week.  Cubic feet per second is in blue, degrees F in red, and acre feet in black - although no station will support all 3 values at once.  Mousing over the graph brings up a vertical bar and provides the time and reported values at that point.  Because station update frequency varies, there is no time range selector like with the snotel sites, at least for now.

The layer is available via the "Water Gauges" checkbox underneath the Snotel layer, or click here to try it.

Tuesday, May 13, 2014

New Fire Activity Layer

As the summer fire season approaches, I decided to take another look at integrating current fire activity with CalTopo.  There's a wealth of government supplied fire information out there, and I've long thought about integrating this into a custom source that plays well with CalTopo's other map layers.  When the MODIS layer I'd previously been using went down and stayed offline for several months, it gave me the nudge I needed.

The old MODIS layer has now been replaced by a "Recent Fire Activity" layer which pulls data from several different sources.

The first of these sources is, no surprise, MODIS detections.  The MODIS system consists of two satellite imagers (Aqua and Terra) that take daily multispectral imagery of the planet; the IR band can be used determine surface temperature and, from that, detect wildfires.  Wide swaths of imagery are taken at approximately 10:30AM and 1:30PM local time; while they don't follow timezones exactly, the East Coast is imaged 3 hours before the West Coast.

The Forest Service's Remote Sensing Applications Center processes MODIS images into wildfire hotspots and makes the locations available at  While there are other sources available that have better global coverage or a faster update time, the RSAC dataset has additional metadata I couldn't find elsewhere, like the actual observed surface temperature.

Edit: Shortly after this post, Southern California saw a bout of fire activity, unfortunately giving me an opportunity to put the RSAC data through its paces.  Due to the time lag between satellite passes and their shapefile updates, I've switched to a new NASA Source which still includes temperature observations.

Starting at a wide zoom, these detections appear as multicolored Xs and *s on the map.  I wanted to convey two different attributes, severity and recency, so I needed to vary both the color and symbology.  Government maps use a red->orange->yellow color gradient to indicate the age of MODIS detections, and I decided to follow that for consistency.  Red spots were detected in the last 12 hours, orange in the last 24 and yellow in the last 36.

The asterisks represent hotter, more likely hotspots, while the Xs represent colder, less probable spots.  Zoom in a little further and each spot is marked with its average temperature.  In busy locations this can get a little crowded:

Each MODIS pixel is at least 1km square, sometimes larger, and the surface temperature is an average across the entire pixel.  So a 122 degree spot could be small, intense hotspot or a large, smoldering fire.  As the temperature drops, the certainty that there's a wildfire there drops; based on this and presumably other factors I don't understand, the RSAC assigns a confidence number to each detection.

At high zoom levels, detections are represented as a circle with additional information.  The X and Y dimensions of the MODIS pixel are averaged to get the circle's radius, so the circle shows approximate coverage size.  The detection date - which may be later than when the hotspot actually started - is shown in Mountain Daylight Time (MDT), regardless of actual timezone.  Temperature and confidence are given on the next line, followed by the satellite that made the detection and the center that processed it (RSAC or GSFC, for the Goddard Space Flight Center).

Another data source I'm pulling in is smoke polygons from NOAA's Office of Satellite and Product Operations.  Color (gray->black) represents smoke severity, while crosshatching is used for current smoke and empty polygons show where the smoke was several hours previously.

The final source is large fire perimeters from GeoMAC.  Unlike the previous two products, which are based on satellite data, fire perimeters are manually reported by an incident's management team.  As a result, they seem slow to update and I'm on the hunt for a more reliable source.  In the meantime, I'm marking perimeters with the last date at which they were updated, in addition to the incident name, so that you know how recent/reliable the data is:

Because fire season is just ramping up, I haven't had the opportunity to test this layer as thoroughly as I'd like.  Normally I wouldn't add it to CalTopo until I'd had the chance to vet it more thoroughly, but since the MODIS layer already on the site was still having issues, it made sense to sub this in as quickly as possible.  Expect things to change, and please send me any feedback or bug reports.