Friday, January 29, 2016

Improved Google Earth Integration

A lot of people use Google Earth to supplement CalTopo's route planning features.  Given that, I'm happy to announce a big step forward in integration between the two.

First, the export menu now has two Earth options - the traditional KML file, and a KML network link, which provides for automatic syncing between Earth and your CalTopo map.  When you open the network link file in Earth, changes get pulled from CalTopo every 10 seconds - move a marker or draw a new line, and boom, it will appear in Earth within seconds.


Data only syncs one direction; changes made in Earth will not be picked up by CalTopo.

The second big change is better layer integration.  I've always provided a few "super overlay" files for loading topos into Earth, but it was an obscure feature limited to a handful of layers.  Now, basic and pro subscribers have access to an improved file, located on the account tab of the account dialog.


Download this file, open it in Earth, and you can pull in a wide variety of CalTopo layers, including previously-inaccessible transparent ones.


A few examples:

Desolation Wilderness with the MapBuilder Overlay layer
Slope angle shading on a popular Tahoe backcountry ski spot

Public/private land boundaries

Sunday, December 6, 2015

PDF Improvements

CalTopo PDFs are now sporting a couple of improvements.

The most noticeable of those is a title input field on the print page:



If filled out, it will show up in a couple places.  On the bottom of the PDF itself:


on the account dialog:


and also in Avenza PDF Maps:


Slightly less noticeable is an improvement in label positioning.  Before, labels were drawn without regard to each other; on a busy map, this resulted in an unreadable mess.  Now, labels will not draw on top of each other.  If two labels conflict, the PDF generator will attempt to move them around (to only few positions; it's not very smart yet), and if that doesn't solve the problem, it will omit the second label to improve readability.

Label 2 automatically floated to the left to avoid conflicts.

Finally, although hard to demonstrate on a screen, MapBuilder layers will now print better.  To create high resolution PDFs, CalTopo starts with a large image and then shrinks it down to fit the page.  With layers that look the same at all zoom levels, like USGS map scans, this works great.  With layers that get redrawn at each zoom level, like OpenStreetMap, shrinks all the lines and text down to the point where they're hard to read.

When you print a MapBuilder layer, the PDF generator will request a tile that's been scaled up by a custom amount so that lines weights and text will come out looking right.  This is only possible on layers produced in-house, but at some point I may ramp up an OpenStreetMap server so that I can do the same thing with OSM maps.

1:24k PDF created without scaling


1:24k PDF with scaling

Tuesday, December 1, 2015

WMS and WMTS: Using CalTopo Layers In ArcGIS

I regularly get asked about using CalTopo layers in desktop programs, often but not always ArcGIS.  There are a number of protocols for accessing seamless map data, and one of them (WMS) has been supported by CalTopo for a while.  However I always kept it closely guarded - if you asked nicely I'd provide the URL, but implore you not to share it with anyone lest a horde of ArcGIS users descend on my server.

With the introduction of paid subscriptions, I can now make this a publicly known feature, along with WMTS support for serving tilesets as-is.  As a note, this is limited to pro accounts, which seems reasonable to me as most people who need to import custom layers into a GIS program are likely paid professionals.

Access is restricted using account-specific API keys.  Click on your account name at the top of the left bar, and then the "Your Account" tab in the dialog.


If you are a pro-level user, you'll see an API access section with links to the WMS and WMTS endpoints.  In this case, ABC123 is my auto-generated (and made up) API key.


If the program you're using supports WMTS, you'll have better luck using that to serve tiles and letting your program combine them into a seamless image.  Because WMS has CalTopo's server compose a new image every time you pan the map, redraws will be slower and can't take advantage of caching.

Both services are provided in spherical mercator projection only (EPSG 3857, 900913, etc).  If you need something else, your GIS program needs to be able to do the reprojecting.

Also note that unlike traditional WMS services, CalTopo's WMS API is only intended to have one layer turned on at a time.  If you turn on multiple layers at once, you may get unexpected results, including no data showing up.

Comprehensive instructions for your program of choice should be available online, but an abbreviated version for ArcGIS and WMTS is shown below.  To start with, on the Add Data dialog, find the GIS Servers folder and choose "Add WMTS Server".


Enter the WMTS URL from your CalTopo account dialog, click Get Layers to verify it, and then OK.


Choose the layer you want to add.


Presto!


As another example, here's the new MBTopo layer rendered in ArcGIS:


Sunday, November 8, 2015

MapBuilder: Pimp Your Topo

I've never been happy with CalTopo's reliance on USGS 7.5' topo scans.  While I expect them to remain the gold standard in backcountry navigation for years to come, in some places they're quite out of date.  Even when they aren't, it's difficult to use them for high level trip planning - zoom out a couple levels and it's hard to see trails or read the text.  Often I'll zoom in to trace out a trail, then zoom out to get the high level picture.

Vogelsang Pass, Yosemite.  At zoom 13, you can barely follow the trails.
I've thought about building my own map layer to address this for years.  The tools are all out there - throw an OpenStreetMap snapshot and some government datasets in a PostGIS database, write some Mapnik styles and put a tile server in front of it all.  The hard part is striking the right visual balance and making it all look good.  Should the map be crowded with details, or sparse and easy to view?  Are public/private land boundaries important, or should they be hidden?  At wider zooms, which features do you keep and which do you progressively phase out?

Much better.
On a backpacking trip this summer, I realized I was thinking about the problem wrong.  CalTopo is all about letting users customize the map they're looking at, and an in-house topo shouldn't be any different.  Instead of creating a single topo layer, I should be letting users design their own.

MapBuilder, the product of that realization, is now here - although it's still in "beta" and under active development.  Once I get things running smoothly, several pre-configured map layers will remain free, while the custom MapBuilder tool will become a pro-level feature.



As with the other custom layers, get started by choosing "Add MapBuilder Layer" under the new layer menu.  A dialog pops up with a bunch of design choices; click Save and your custom layer will show up in the dropdown next to all the others.


There are also two pre-configured layers in the Topographic Maps section of the standard layer dropdown, although their composition may be subject to change.  MB Topo looks like a basic topo map, while MB Aerial uses an aerial imagery background and drops some of the clutter, like peak names and park boundaries.

Anyway, enough with the talk, here's a quick tour.

MB Topo view of the Marin Headlands

MB Aerial rendering of the same spot

MB Topo rendering of Mt Olympus, WA
Custom Grand Canyon map that emphasizes trails.
Custom high-level planning map of Cascade Pass, WA.
Custom map of Logan Pass, Glacier National Park, MT

Some caveats:
  • The data I'm using for relief shading has some issues with wide zoom levels.  It should be fixed in a week, but until then North-facing slopes may look rough.
  • OpenStreetMap trail data includes some common cross-counry travel routes, like the Whitney MR and the DC and Emmons routes up Rainier.  So far I haven't figured out how to distinguish them from actual trails.
  • The PDF generator creates high resolution maps by zooming in and then shrinking everything down.  With layers that change at each zoom level, this makes text and features really small.  I hope to have a solution soon, so that MapBuilder layers will print acceptably.
  • Since this is under active development, existing maps may cease to work or suddenly start looking different.
  • Remember, once I get the bugs worked out, custom layers will become a pro-level feature, but several preconfigured layers will remain available.

Saturday, October 24, 2015

Subscriptions Go Live

While I don't expect this news to be greeted with thundering applause, CalTopo subscriptions are now live.  Based on feedback from y'all, I moved to a two-tier pricing model:

You can still create, share and print maps for free, but prints will be limited to 5 pages of 8.5x11 and KMZ and MBTiles exports will be size limited.

At the moment, the Pro level is simply everything the Basic level has, but bigger - maybe I should have called it "super size" instead.  Unless you really need large-format PDFs or large scale KMZ or MBTiles exports, I realize it may be a tough sell for $50 at the moment.  However there are some additional features slated for the Pro tier that I'm pretty excited about, and some of which will carry significantly increased costs, and I don't want to be stuck needing to raise prices in six months.

For the detail-oriented, there a few sneaker items in that table:

  • PDFs can now go to 48" instead of just 36".  Perfect for those with access to office plotters and ink purchased by the barrel.
  • The maximum PDF length is now 40 pages, but multipage PDFs are limited to 8.5x11 and 11x17.  If you absolutely need a 5 page 24"x36" PDF, you'll need to make five separate documents and combine them on your own.  That keeps my server much more sane.
  • The default PDF retention limit is now 7 days.  Most people don't need a linkable PDF longer than that, and PDF data was becoming a significant portion of my total DB size.
  • KMZs now go to 200 images instead of 100.  Be aware that most Garmin GPSs won't handle more than 100 images across all KMZs loaded on the device, because Garmin wants you to buy BirdsEye imagery instead.
  • MBTiles exports are now possible.  This has become the go-to tile storage format for smartphones, and I'm excited to offer it.  More on that in another post, but if you use an app that already licenses my map layers, such as Gaia or Backcountry Navigator Pro, you're generally better off downloading maps directly through them.
In addition to those user-facing features, I also upgraded (again) to a new server with twice the RAM.  For the second time in a year, CalTopo had started experiencing short, intermittent outages from running out of memory during heavy usage.

I've tried to keep nagging to a minimum and to avoid full-page "Upgrade Now!" popups.  I'm firmly convinced that CalTopo has a solid value proposition and that the site itself is my best sales tool.  Unavailable options are grayed out and have an upgrade notice nearby, but otherwise, there's plenty of opportunity for new users to discover why the site is so valuable, instead of me beating them over the head with it.


If you are signed in, and are not a subscriber, you will also see an upgrade link in the top menu:


To change your account status, or simply find out more about it, go to https://caltopo.com/join or click on your account in the left bar, choose the "Your Account" tab, and then the "Change Status" button.  You'll see something like this:


Payments are handled by Stripe; your credit card number goes straight to their server and I never handle or know it (other than the last 4):


Any problems, please drop me a line at info@caltopo.com.  SAR members and people who have previously donated should contact me directly for a redemption code.

Monday, October 19, 2015

HTTPS Support

In preparation for subscriptions and credit card handling, CalTopo now supports secure https connections.  Because all requests from an https page must also be in https, there are a few features that don't work quite right yet, most noticeably the land management, fire activity and fire history layers, but that's all I've found so far.  If you run into other issues, please let me know.


That concludes this public service announcement.

Wednesday, September 9, 2015

CalTopo Goes Commercial

I've spent a lot of time lately thinking about the future of CalTopo.  While I'd love to claim that I started with some kind of genius original vision, the truth is that it started as an offshoot of a hobby project I'd been working on for SAR, and grew organically from there.  As with any software project, the work started out fun and gradually became more of a drag.  Now much of my time is spent answering emails, fixing bugs, and dealing with operational issues.

In short, the first 10 hours or so a week I spend on CalTopo have all the excitement of a regular job, but without the upside of getting paid.  I only get to do the fun stuff after I eat my broccoli, and many weeks that never happens.

Happier times, when CalTopo's landing page didn't have a map viewer, you had to create a map before you could draw anything, and I didn't have any support work.

After devoting my free time almost entirely to CalTopo for a number of years, I've been getting restless.  The past year saw me delve into researching the impact of terrain on lost person behavior in SAR.  While the results may seem obvious, an entire generation of search managers has been trained using the aeronautical/maritime model of gridded area searching with little regard to terrain.  The Department of Homeland Security, which fancies itself as owning inland SAR in the US, is still heavily invested in that model, as is the current crop of textbooks.  My research project is a tiny drop in the sea of effort required to restore some sanity to the way people are found, and I would like to do more.

Another consideration is my AWS hosting bill.  August cost me $2200; rounding a bit, the costs were roughly:

  • $250 to host map tiles on S3
  • $200 to host the caltopo.com and sartopo.com servers
  • $100 of random bits and pieces, such as EBS snapshots full of GIS data that I need to keep around
  • $1400 in data charges, mostly serving map tiles


Yikes!

There are a few smartphone apps that license the map tiles from me; they form the bulk of the $1400 in usage costs, and also pay me just enough to subsidize the entire AWS bill.  However if they were to pull out I'd be up a financial creek: beyond the fixed hosting costs, CalTopo users alone account for $100 a month worth of map data.  This is not some forum that can be run on a cheap shared host, and while I greatly appreciate the various donations people have made, they fall well short of the site's actual monthly operating costs.

The costs involved with developing and running CalTopo - both real and opportunity, financial and time wise - have been significant.  While I still cry a little inside every time I price out a 4x4 Sprinter camper van, I don't regret the hours and funds plowed into it.  However, looking to the future, how long can I continue channeling such a large part of my life towards CalTopo?

Yeah, Baby!  If I could go back in time, would I trade CalTopo for one of these bad boys in my driveway?  Decisions, decisions.

I don't know what the answer is, but it's certainly not another 5 years.  Even ignoring the pull of other projects, I've simply been spending too much time locked away behind a desk.  My own sanity requires more weekends spent recreating rather than working.

As I see it, there are basically three options:

Keep the current model and let the site stagnate.  I'll occasionally be able to make time for improvements, but the pace will slow and large features like auto-routing along trails are unlikely to get implemented.  If map licensing fees drop off, either shut down or eat the $7k/yr in unavoidable hosting costs.

Open source the code and start a nonprofit to cover costs.  I tried the open source approach with early versions of the code, and no one ever contributed anything (literally, which is why I was able to return to a closed source approach).  With the site's improved popularity it may be a different story now, but I'm not convinced that the open source approach works well with hosted sites, as opposed to software that end-users run locally, or server frameworks like Apache.

In addition to code improvements, there are also the hosting costs to worry about.  I've seen plenty of nonprofits struggle for cash; the most effective ones either have strong local appeal or some kind of national advocacy message.  A backcountry mapping / SAR nonprofit wold have neither, and it's unlikely that I would be able to do more than struggle to cover each year's hosting costs.  That said, if anyone has a pile of money and a burning desire to help the outdoor and SAR communities with it, drop me a line at info@caltopo.com.

Get serious about running the site like a business.  Beyond hosting costs, try to make enough to justify continued development and investment into the site.

After a lot of thought, I've decided to go with option 3.  While there was some hesitation leading up to the decision, now that it's made, I'm pretty excited about it.  A sharper focus on commercially viable features will ultimately mean a better site for users, with me forced to spend more time on documentation and UI improvements rather than chasing shiny new ideas.  It opens up the potential to include paid map layers, such as PDFs with higher resolution aerial imagery.  And, hopefully, it will allow me to invest more hours developing new features.

The "shared maps" layer was originally driven by commercial ambitions, but now that it exists, I think it's a great feature in its own right.

Understandably, I don't expect all the users out there to share my enthusiasm.  Since I'm explaining why I need to make CalTopo commercially viable, I also want to clarify a few things:

  • CalTopo will continue to be usable as a free site; it will not be hidden wholesale behind a paywall.  As a user of the site myself, I want to be able to bring a map up on a friend's computer without having to sign in.
  • CalTopo and SARTopo will continue to be free to first responders who use them professionally, such as SAR and wildland fire.
  • Billing will be simple and transparent.  You won't have to enter a card number to start a free trial, and  won't have to time your cancelation with the end of a billing period.  You can cancel with a single click, no annoying retention process.
  • You can keep your data.  You don't have to worry about losing any of your saved maps or PDFs if you cancel your subscription.
  • If you've already made a donation, you can request to have that credited towards your account.

I'm still working out the exact details and will lay them out in another post.  However, my current thinking is a $40/year subscription that includes the following:
  • Unlimited private maps.  Free users will be limited to 5 private maps, although they can create as many publicly shared ones as they want.  If you already have more than 5 private maps, you can keep them, you just can't save any new ones until you get the total under 5.
  • Save PDF URLs for up to a year.  For free accounts, URLs will expire after 30 days.
  • Large-format PDFs.  Free accounts will be limited to 11x17.
Edit: I should have made it clearer that this is simply one option.  I'd also consider something that's cheaper but encompasses more of the site; see comment below.

One feature I'm still contemplating is a smaller cap on the number of pages a PDF can have (current limit is 15 across the board).  There may also be some additional paid PDF management functions, such as the ability to hide the QR code or point it at a custom URL.

So . . . what say ye?  What's the best price / set of features that will give me a functional free site and happy paying customers?