Friday, December 12, 2014

Feature Week: Print to KMZ

I've received a few emails asking where the KMZ link in the left bar went.  It's been replaced by a print-to-KMZ option:


Why?  Because instead of just map layers, KMZ exports can now have markers and lines "printed" on them.  These are not traditional KML placemarks - for that, use the standard Export option.  Instead, they're rendered as graphics along with the map.  This can look a little fuzzy in Google Earth:



However, it really shines when used as a Garmin custom map.  You can draw far more lines on the map than most GPSs can handle as tracks or routes, and it allows you to create more colorfully marked up GPS maps.  Example screenshot below:


Thursday, December 11, 2014

Feature Week: Bulk Operations

If you've ever wanted to change the color on a number of lines at once, the new bulk operations feature will nicely complement folders.  The Markers and Shapes boxes, along with folders, now have a Bulk Ops link at the bottom.


Bulk operations can only be run on one object type at a time, so if you click Bulk Ops on a folder with both Markers and Shapes, you'll have to choose.


Once that's done, you'll see a full page dialog listing each object with some options at the bottom.  Click or shift-click to select a group of objects.


The Print Individual Maps option is only available for saved maps.  It will open a multipage PDF in a new browser window, with one page per object.  Each page is automatically centered on the object and scaled to fit, with a minimum scale as specified.  The default is 1:12k, which strikes a nice balance between capturing small map details and not losing context.

Change Attributes will bring up a second full page dialog, similar to the object's details dialog.  All fields start off blank, any changes made will be propagated to the selected objects.  Beyond setting marker and line styles, you can move objects between folders en masse.


Delete Objects will obviously delete the selected objects, although not without bringing up a confirmation window.

Wednesday, December 10, 2014

Feature Week: Folders and WYSIWYG Printing

Over the years I've had a number of users ask for a way to organize markers, lines and polygons within a map.  There have also been requests for a way to print only some objects, or to print markers with labels and shapes without, and so on.  Today's post covers a new way to do just that: folders.

Start by adding a folder to the map the same way you would any other object: right-click on the map background or click + Add New Object in the left bar.


The details dialog for markers, lines and polygons now has a folder dropdown.  Choose a folder at creation time, or move objects between folders the same way you would set any other property.


Individual folders will show up as boxes in the left bar alongside other object types.  Objects without folders will remain in their object type box, e.g. Markers.


As with the Markers and Shapes boxes, you can use the checkbox to show or hide all objects within a folder.  If you want to hide labels for a folder, perhaps because they're too noisy, the settings box now has a third, custom label option.


When set to "some", each folder gets a label toggle at the bottom.  Click on it and "Labels Shown" will change to "Labels Hidden".


Alongside these changes, printing has moved to a what-you-see-is-what-you-get (WYSIWYG) model: what you see on the screen is what you get when you print a map.   If an object is visible when you click print, it will show up on the print.  If it's hidden, it won't.  If it has a visible label, that label will show up on the print.  If the label is hidden, it won't.

Because this requires a form POST, I can now only offer the "share or save this configuration" link for a blank map, not one that has objects on it.  However I think the extra flexibility when printing is worth the cost.

Monday, December 8, 2014

Feature Week: Automatic Syncing and Collaboration

The past several weeks have seen significant behind-the-scenes performance upgrades to CalTopo, including mass migration of lines and polygons to a more efficient storage format.  Some of these changes were not without risk, so if anything looks off, please don't hesitate to let me know.

Only time will tell, but it looks like I'm out of the woods on performance work, and that has allowed me to roll out some long-deferred changes.  To celebrate, I'm going to run a series of posts highlighting new features.  The first one is automatic syncing and collaboration.

Without running afoul of trademark law, I've often tried to describe CalTopo as being like "Google Docs for maps".  While not as fully featured as a desktop program, from the beginning it enabled collaborative sharing and editing of maps in a way that simply wasn't possible with software like Nat Geo's now-discontinued Topo!.  However, for performance reasons, true Google Docs style "everyone sees all changes instantly" editing remained locked behind an optional checkbox in the sharing dialog.

No longer.  Starting now, saved CalTopo maps will automatically pick up on changes as they are made by other users.  Unlike Google Docs, there will still be a bit of a delay - from 5 seconds for actively edited maps up to 30 seconds for maps that have not been changed in the last 10 minutes.  After a couple weeks I may be able to dial this down depending on server load.

So how do you start this shared editing?  Time for  a quick refresher.

  1. The map needs to be saved (URL ends with /map?id=ABCD).  If you're working on an unsaved map, click the red Save link at the top of the left bar.
  2. Click Share.
  3. In the permissions dialog that comes up, either give write access to all users who know the map's URL, or set a password and give write access to users who know the password.
  4. Send the map's URL (e.g. http://caltopo.com/map?id=3L6J) to a friend.
  5. If you set a password, they will need to click on the Share link and enter it.

That's it.  Happy collaborative editing.

Friday, November 7, 2014

Mo Users, Mo Problems

Over the past several weeks, I've been getting progressively more frequent emails from the Binary Canary monitoring service letting me know that CalTopo was down.  It was sometimes unresponsive for extended periods of time, but since it generally came back on its own, I did what any good developer would do and ignored the problem until it reached red alert levels.
After spending 2 days a very long ways away from cell reception for SAR, I came back to a flurry of emails, including another Binary Canary outage warning - and CalTopo still appeared to be down, 5 hours later.  This was the tipping point; something had to be done.

Throwing together some changes and deploying them with a half-assed amount of testing was something, so that's what I did.  While it solved the performance issues, it broke imports and exports - something which CalTopo users let me know in short order.  Now that the fire drills are over, here's a quick recap of the issues.

One problem was in the way I sent line/polygon points from the server to the browser.  Instead of using a numerical array, each line point was a separate object.  This isn't a big deal in and of itself, but the JSON library I'm using to serialize everything created a heavyweight map object for each lightweight waypoint object, resulting in a huge amount of object churning relative to the memory available.  Points are now sent as an array, which doesn't have that overhead.

The second problem was - and possibly still is - with supplying cursor elevation.  Because I push elevation data to the client as a plain-text array of numbers, the 256px tiles are significantly larger than PNG or JPG map tiles.  As an added bonus, because elevation data went through the same server-side process as shapes and markers, the browser was instructed not to cache any of it.  Every page load generated a flurry of elevation requests.

One small checkbox, one giant bill.
Since turning the "Include Elevation" checkbox on by default, it turns out I was spending $150/month just to send elevation data to clients.  Oops!  I have a rather hefty Amazon bill, most of which is reimbursed by smartphone apps that license my maps.  Amazon makes it difficult to separate EC2 data charges (my server) from S3 data charges (map tiles), which is why a number several times larger than my monthly beer budget had slipped by unnoticed.

Now that elevation data is getting cached, I'll give it a week or two and decide whether I need to go back to having the checkbox disabled by default; the answer will probably be yes.

What this incident has made clear is that I need to find a way to make CalTopo commercially viable.  I've asked people to send donations to BAMRU because I thought that as a 501(c)3, it would encourage more liberal giving.  However, even if the donations BAMRU receives were funneled directly into operating the site, they would still be insufficient to cover my monthly operating costs.  Fortunately there are commercial map licensees for that, but while they pay me enough to cover my fixed costs, they're not really not a profit center.

When asked about my plans for commercialization, I've always maintained that while I don't want to cover the site with ads or turn it into useless baitware, I'd rather offer a better experience at a reasonable price than a mediocre one for free.  If it allows me to devote more time to the site and provide a better feature set, some kind of paid model would be an all around win.

After a lot of delay, I think it's time to go down that road.  More details will be provided as I figure them out, but I think the following is a safe high-level picture:
  • Most of the site will always be free.  From a purely self-interested perspective, I want to be able to hop on someone's laptop and sketch out a map without having to log in, verify my account, etc.
  • Any paid features would be part of a subscription rather than pay-per-use.  A number of people have suggested pay-per-print, but that seems rather petty; I'd prefer that people not have to ration their usage.
  • I'll try to align paid features with my cost structure, i.e. the features that cost me more time or server resources to support will be the ones I charge for.
  • Everything, including offline map data, will continue to be free for SAR and other first responders.
  • I'll be selling related items like GPS maps and 3D posters as an additional funding source.
So with that in mind, what are some things that might become part of a subscription?  Nothing is settled, but ideas that come to mind include large PDF sizes like 24x36, PDF links that last longer before expiring, WMS map access and having elevation data show up by default.  I'd also like to integrate non-free map sources (i.e. I need to pay the map provider), and a subscription seems like the logical way to tackle that.

The goal is to devote more energy to CalTopo, not to let it wither while I try to wring money from an ever-shrinking user base - though that seems to be a popular approach, too.  Paid features are an experiment, and if they turn out to be a bad idea then I'll drop them.  Changes will be neither drastic or immediate, and you don't need to worry that PDF generation or some other critical feature will suddenly wind up behind a paywall.

All of that said, I'm open to suggestions.

Monday, October 6, 2014

Daily Dose of MODIS

I've written previously about the MODIS satellites and their constant imaging of the planet; CalTopo's fire activity layer is derived from their infrared sensors, with a lot of post-processing by the federal government.  There's now a new pair of layers that make use of the visible-spectrum sensors: current and archived MODIS imagery.


The Latest MODIS layer can be found under the Aerial Imagery group.  Imagery is acquired twice a day, but note that the first pass may not happen until the afternoon.

All quiet in California, but a little foggy on the coast.

The layer has a nominal resolution of 250 meters / pixel, but since the imagery is often not tack sharp, usable resolution will be less:

Not quite enough to read license plates.

The latest image may be covered with clouds, or the area you're interested in may be especially blurry if it was at the edge of an imaging strip.  That's where the archived imagery comes in, as a new configurable layer option:


Choose which pass you want (the satellites are named Aqua and Terra), the date you want imagery from - today, yesterday, or any day back to May 2012 - and you're off and running.


You may be thinking "so this is a fun toy, but how does such a low resolution image help with backcountry trip planning?"  To answer that question, lets dial the clock back a couple weeks to September 16, when the King Fire was going strong:


CalTopo's fire activity layer pulls in smoke polygons from a separate source, but there's nothing like seeing it with your own eyes.

As another exercise, a light storm blew through the Sierra shortly before some friends headed to Desolation Wilderness last fall.  They asked me how much snow to expect, and I erroneously under guessed as Truckee had already melted out several days prior.  If only they'd had easy access to MODIS imagery, it would have been obvious that their destination was still coated in a thin layer of the white stuff.  As an added bonus, you can make out smoke plumes from controlled burns along the western slope of the Sierra:


This has barely gone live and I'm already in the habit of checking it once a day, just to see if anything interesting is going on.

Monday, September 8, 2014

End to End: Closed Contour Yosemite Maps

A few years ago, I ran across Dan Cervelli's excellent Closed Contour map of the Sierra.  At the time I was just getting started with CalTopo and learning my way around GIS, and Closed Contour really opened my eyes to the cool stuff you can do with Mapnik.  I had no idea it was so simple to take some shapefile data, add a few rendering commands, and then boom, beautiful looking maps.  At least, the tools are that simple - obviously there's still a lot of talent and hard work involved in producing the final product.

The original Closed Contour map was built on a projection and tiling system incompatible with Google Maps, so integration into the universe of sites and apps using that as a de-facto standard was not possible.  Dan has a new Yosemite map built around this standard, and I thought it would make a great end-to-end demo of some things CalTopo can do with "web mercator" tile sets.

Step 1: Add a Custom Layer

The more maps, the better - CalTopo is built around the idea of combining different map layers into a single coherent picture.  Unfortunately, I have to balance supplying layers of regional or special interest against providing a reasonably compact, easy to navigate UI.  The middle ground I've settled on is to make it easy to add custom layers from a tile URL or WMS address.  You can also choose from a preset list of useful layers, which Closed Contour has now joined:


Closed Contour now appears in the layer dropdown alongside the standard CalTopo layers


and provides a snazzy backdrop for some route planning, complete with trail mileages


You can also skip these steps and just visit http://bit.ly/1pIA98Q to take a look.

Step 2: Make a PDF

I've already covered printing multipage PDF map packs and loading Geospatial PDFs onto your smartphone, so lets just jump ahead to the results:

The Half Dome hike fits onto a single 1:24k scale 8.5x11 print.  Compare Closed Contour, above, to the USGS 7.5' topo below:


I can also load this into Avenza PDF Maps to plot my location and record tracks when out in the field:


Step 3: GPS and Google Earth

Both Google Earth and newer Garmin GPSs know how to read layers exported from CalTopo as KMZ files.  So lets go make one:



This gives me a better GPS map of Yosemite than money can buy:

 

And some pretty nice 3-D visualizations: