Monday, December 3, 2012

Garmin Custom Maps

Newer Garmin GPSs (including the 62, Oregon, Colorado and Montana series) allow you to use KMZ (Google Earth) files as custom map layers.  You can now export CalTopo's map layers and use them with these devices.

  1. Go to http://caltopo.com/kmz.html or click Print and then choose KMZ.
  2. Navigate to the area you want to export
  3. Set the map background using the dropdown at the top right.  You can combine layers as usual, e.g. shaded relief + topo + slope shading
  4. If needed, click "fit selection to screen" and drag the rectangle to cover the area you want to export
  5. Choose a zoom level, remembering that you can only fit 100 images total on the GPS
  6. Click "Download KMZ", or click "Connect to GPS" to send the file directly to a newer Garmin GPS using the Communicator plugin.

Exporting avalanche slope shading to a KMZ file.
  
To load a KMZ file into your GPS (if you didn't transfer it directly using the plugin):

  1. Plug in your GPS using USB.  It should show up as a removeable drive, just like a USB thumb drive
  2. Create a \Garmin\CustomMaps\ directory on the GPS, if it doesn't already exist
  3. Copy the .kmz file into \Garmin\CustomMaps\
You may also have to turn on the Custom Maps layer on your GPS.

Nationwide Google Earth Coverage

I've supplemented the existing KML export capability with a new feature that dynamically generates KML files as you fly around in Google Earth.  This means that you can get nationwide coverage from a single KML file, without needing export small areas on a one-off basis.

Try it!

Click here for USGS Topo maps
Click here for USFS Road and Trail maps

Forest Service maps on top of USGS topos, with large-scale coverage.

Please note that this will become a paid feature in early 2013, but is free for now while I finish development.

Printable PDFs

CalTopo has always allowed you to print maps directly from your web browser.  That feature isn't going away, but it has some disadvantages:

  1. Poor Browser Support.  You can make nice prints with Chrome, but Firefox doesn't respect CSS print margins and if you're not careful the map can easily bleed over into a second page.  IE prints the URL and title at the top of the page, with no way for the user to turn them off.
  2. Low Resolution.  On my computer at least, setting the map size in inches gives you a print resolution of only 100ppi.
  3. Fixed Scales.  There are a limited number of zoom levels available, and the scale doubles with each one.  Often zoom 14 is too broad and 15 is too narrow; there's also no way to print a 1:24K map.
In response to these issues, CalTopo now offers server-side PDF generation.  Pick your layers, draw a rectangle on the screen, and hit print.  You'll get back a high resolution PDF that will print exactly as it looks, no muss, no fuss.


At this time, PDF generation can only be used to print CalTopo's base layers.  It can't be used to print Google's layers or maps from other sources.  It also won't show waypoints and tracks from saved maps, although I hope to add this at some point in the near future.

Try it by going to http://caltopo.com/print.html, or by clicking on the "printable PDFs" link inside the standard map page's print mode.  Please note that this will become a paid feature in early 2013, but is free for now while I finish development.

Friday, September 14, 2012

Custom Map Layers

Taking an image and turning it into a custom map layer through georeferencing is something that was previously supported - poorly - as a SAR-only feature.  Storing and rotating the image server side took too much memory and wouldn't scale well to a broad public user base.  Now that all 3 major browsers support CSS image rotation, I've taken another crack at implementing it as a browser-only solution.

There are a couple restrictions on the type of image you can use.  First, it has to be hosted online somewhere - web server, flikr account, whatever.  CalTopo won't upload and host the image for you.  Second, it needs to be drawn to scale.  CalTopo will scale and rotate an image, but it won't do any warping.

Custom layers are shown on the left nav bar below markers and shapes.  Click New Layer to get started.


First, supply the image URL and click Set.


The image should appear on your screen, with the bottom left and upper right corners aligned to the corresponding corners of the map viewer.  You can mark the two image points, and their corresponding  real-world coordinates, by right-clicking on the map background and selecting Georeference.


Use the opacity slider on the custom layer dialog to hide the image and show the underlying map.  Once you've marked both point/coordinate pairs, you can fine tune the results by continuing to click on the map or through the dialog.  When done, click save.


Once the layer has been created, it appears in the overlay dropdown along with all the other layers.  You can't use the custom layer as a base layer.


If you're working on named map, the custom layer will be saved along with that map.  If you're just using the regular map browser, the custom layer will be added on to the URL hash string, so that you can bookmark it or send a link to other people without saving anything to CalTopo.  As an example, here's the Huddart County Park map with shaded relief and contours.   Click here to see it in person.


Thursday, July 26, 2012

Canada, eh?

I'm happy to announce that a topo layer for the canadian rockies is finally live.  The "nrcan" WMS source has always been available from the candian government, but I've wanted to create my own tiled source as well.  Why?

There are 3 sets of topo maps available from the canadian government:
  • Toporama is a WMS layer hosted by Natural Resources Canada, and appears as "nrcan" on CalTopo.com.  It's a vector based layer - all the trails, contours, water features, etc. are stored as lines in a database and then turned into an image using a renderer like Mapnik.
  • CanMatrix is scans of the original 1:50k Canadian topos.  These are what the current layer is based on.  Scan quality only supports going down to zoom 15.
  • CanTopo is a newer, high-resolution round of topo maps.  They appear to be computer-generated, probably from the same vector data that feeds Toporama, but coverage is currently quite sparse.  Image quality supports going to zoom 16; as coverage increases, I will expand the USGS 7.5' layer to include these.
Being a WMS layer, Toporama isn't as speedy as I'd like.  Although it's better than the USGS' vector data, side-by-side comparisons show that there are still a lot of missing trails and forest roads compared to the older paper maps.  Some of the data leaves you scratching your head for context, like short unlabeled trails that start and stop in the middle of nowhere.  And, being WMS, it's not compatible with some of the tiled-dataset tools I'm working on like Google Earth Export.  Not that you couldn't make it work, but it's more development effort.

CanTopo would be the ideal dataset to build a new map layer off, but it doesn't exist for most of the country yet.  The maps I've spot checked look dead on compared to the old paper ones.  I suspect the slow progress is due to the government checking/updating all the vector data before producing the map; perhaps when the full series is available the Toporama data will be equally comprehensive.

That leaves CanMatrix, which works fine but suffers from a couple imperfections.  Some of the maps are NAD27 and others are WGS84; historical NAD27 versions of the newer maps don't seem to be available.  This results in small coverage gaps at the border between maps with different datums.  Many of the older maps, particularly getting up north into the Yukon, are grayscale only.  And finally, the accuracy seems way off compared to the USGS 7.5' series.

I kept iterating on the rendering process because the maps didn't line up well with Google's terrain/satellite layers.  Beyond the datum, if you get the projection wrong it can cause parts of the map to be distorted while others are dead-on, and I thought this was my problem.  Eventually I realized that some of the maps just have massive survey errors (and by massive, I mean 150' - not really noticeable unless overlaying against a reference map).  This results in some noticeable misalignment where map quads meet.

For the US, the difference between the new "US Topo" vector maps and the older paper maps is night and day.  Because Canada's paper maps are worse and vector maps are better, the difference is much less pronounced.  Because of this I decided to only render out the rockies / western coast, where the road and trail information on the paper maps is still critical, and skip the flatter lands in the rest of the country.  If there's interest, though, I can look into expanding coverage.

Wednesday, April 25, 2012

Exporting Map Tiles to Google Earth

Sometimes a flat map just doesn't cut it.  You want to be able to rotate it, skew it, project it onto real-world terrain.  Google Earth is great for this, but their aerial imagery only takes you so far in the backcountry.  You need trails, contour lines, streams - and you need to be able to match what you're seeing with the paper map you'll take into the field.

The view past Half Dome into Tenaya Canyon, Yosemite.
CalTopo now solves this problem by letting you export map layers to Google Earth with only a few clicks.  Simply start the CalTopo map browser, navigate to the area you'd like to capture, and click Export to Google Earth in the left navbar.  Select the map layer you'd like, hit Export, and open the downloaded KML file.


A few things to know:

  • There is a rendering bug with Google Earth that causes transparent images to occasionally show up with black backgrounds.  This unfortunately affects the slope shading and contour layers; a solution is in the works.
  • An export is limited to the current screen, but Google Earth will seamlessly patch multiple exports together.
  • For performance reasons, resolution is limited based on the map bounds.  Zooming in on a map will get you a higher resolution export that covers a smaller area.

Death Valley, CA
Tahoe City, CA circa 1930
If you find a map that looks especially cool when loaded in Google Earth, please post it in the comments section.

Tuesday, April 10, 2012

End of an Era: Lights Out at Terraserver/MSRMaps

MSRMaps, the site formerly known as Terraserver, is shutting down at the end of the month.  It was a groundbreaking service when it first came out, and I lost many hours poring over slow-loading aerial images and topo maps on it.  In some ways I'm not surprised as even today the site could hardly be accused of being speedy, and it was clearly not Microsoft's top priority.  However its absence will still leave a large hole in the online topo field, and it seems like a good opportunity to look at the current options.

MyTopo used to be the go-to source for web-compatible topo maps, but Trimble bought them, and after 9 months of business as usual, decided it was time to take their ball and go home.  You can still include their map tiles in a web app if you (1) pay them a bundle of money or (2) can live with watermarks that take up half the image.  I'm not sure if using them in a smartphone app is even an option.  It's a shame because MyTopo's map layer had a couple things going for it:
  • Seamlessly integrated Forest Service maps.  They made some bold claims about updating maps to reflect new USFS roads and trails, but in reality it seems like they took the Primary Base Series raster maps and added in vegetation shading, which is still no easy task.
  • Canada and Alaska coverage.
  • Normalized colors across scans.  Each quad used exactly the same color for white, green, brown, etc.
ESRI makes a number of maps available via WMS and a web tile scheme, including a topo layer based on scanned USGS maps.  All their licensing information seems directed at ArcGIS end-users and not websites, but it says nothing to indicate that you can't pull it into a web map.  These maps cover Alaska but not Canada; unfortunately the image quality is lower than I'd like to see.  They only go to zoom 15 v. CalTopo's 16, and even then you can see serious artifacting that suggests the images were over-compressed.

The National Map is run by the USGS and offers up a mishmash of layers via a mishmash of access schemes.  Much like Terraserver, they have the older low-res DRG scans available through WMS.  Although the scans leave a lot to be desired, this service seems like a good Terraserver stand-in for now.

And of course there's my favorite high-res map layer, CalTopo.  The USGS has finished high-res scans for all of the continental US except MA, so it's time to take another pass and fix some of the coverage gaps I ran into in January.  If you'd like to include any the CalTopo map layers in your web site / iPhone app / whatever, send me an email and we can work out the details.

Monday, April 9, 2012

Working with US Topo Maps

The USGS has stopped updating to their traditional 7.5' paper quads, replacing them with a new product called US Topo, a reincarnation of their Digital Map - Beta product.  This seems like a never ending source of confusion (I'm looking for a US Topo map.  You mean a map of the US?  No, a US Topo.  Yeah, like one that covers the whole country?), but it is what it is.

You can still buy paper copies of the US Topo maps, so don't think of the new approach as a shift from paper to digital, but rather as a change in the way maps are produced.  The traditional paper maps were hand-built from survey data and stereoscopic imaging; any computer involvement was more along the lines of photoshop than a GIS application.  The new maps combine a number of layers, some entirely computer generated, into a single PDF.  So with the right tools, you can switch between viewing trails on a traditional white-green background and viewing them on an aerial image.

A US Topo map with all layers turned on.


Having now built out a nationwide seamless tile set based on these new PDFs, I have some comments on them.  Although these comments are mostly of the negative variety, I actually think these maps hold a lot of promise, but that they need a little refining.  So on with the issues . . .

Poor Toolchain.  You can open a US Topo map in any PDF viewer, but it's virtually impossible to turn individual layers on and off.  The USGS directs you to TerraGo's windows-only toolbar, and I sure hope TerraGo paid them a bundle of cash because not only is their toolbar a pain to work with, it comes with an onerous registration and download process.  When I made the mistake of downloading it onto a Mac, I actually had to stick the installer onto a USB drive to move it to a windows machine because my 1 free download had been used up.  I don't know who exactly is clamoring to pirate a GeoPDF plugin . . .

I use GDAL to slice individual quads into the tile pyramid used by web maps, and its GeoPDF driver has no support for toggling layer visibility off and on.  I was about to give up after a lof of web searching when I stumbled on the missing link - although referred to colloquially as layers, the various map elements are implemented as optional content according to the PDF spec.  Armed with this knowledge, I  dove into C++ code for the first time in 15 years and modified pdfdataset.cpp to deal with optional content.

Inconsistent Layers.  US Topo PDFs have a lot of layers.  They're grouped hierarchically, with some of the high level groupings described here.  Unfortunately when you get beyond the spec, all bets are off.  I was hoping to create a white-green woodland base layer in addition the imagery base layer, but it's not present on all the maps - some you can roll back to the traditional look, some you can't.  Some have glacier shading, some don't.  Even when maps have the same layers they're not always named consistently, which doesn't matter for a casual user but is a killer when automating a process across 55,000 maps.  I'd really like to see the USGS expand the list of mandatory layers included in the spec.

Odd Rendering Choices.  The USGS has some beautiful sample maps and the US Topos certainly shine in some situations, but they're practically unusable in others.  The traditional brown contour color is almost impossible to distinguish against some imagery backgrounds; I decided to omit it and use my own orange-brown lines instead.  Some streams and trails are so faint that they virtually disappear if the map gets reprojected.  The biggest annoyance, however, is the labels: the white drop shadow makes them go from huge to unreadable over the course of about 2 zoom levels; water features use a baby-blue color that is even harder to read over the drop shadow than the standard black.  Simple black lettering from the traditional topo won't show up well on aerial images, but I wish they'd found a better solution than the one they went with.

Missing Data.  This is the part that really blows me away.  Google's Hybrid map has great coverage for urban areas; I was hoping that US Topos would be a substitute for backcountry roads and trails.  Unfortunately a number of trails are flat-out missing and some forest service roads aren't properly rendered, while the urban features I care least about are well cataloged.  As an example, the timberline post office is prominently labeled on Mt Hood, but neither Cascade River Rd nor the Cascade Pass Trail can be found on the map despite being one of the most scenic short hikes in the Pacific NW.  Nearby sits this very confusing road intersection which violates 100 years of the USGS' own mapping conventions by displaying 4WD roads as single dashed lines.  The map key gives no indication of whether double-dashed roads are gentler or rougher than single-dashed ones, or how to distinguish these roads v. the hiking trails from which their symbol was co-opted.

Although these maps look really cool, I wouldn't throw away the old-school "historic" maps until the roads and trails are better updated and every PDF gets a green/white land cover layer.

Wednesday, February 8, 2012

Avalanche Slope Analysis

CalTopo has a slope analysis layer that can shade slopes by angle and aspect.  You may or may not find it useful for planning safe travel routes in avalanche-prone terrain, and I make no claims that it is either accurate or useful for this purpose.  However, if you do decide to use it for route selection, it is imperative to understand that this is just one of many tools you should use and that it has serious limitations which can get you killed if you don't understand them.

Although it may look tempting, you cannot transfer an avalanche danger rose into CalTopo and expect that the result will keep you safe.  Remember that an avalanche forecast is a broad assessment covering a large range of terrain, and is not intended to be used with the pinpoint accuracy that CalTopo's aspect shading provides.  I'll list a few known limitations, but there may be others, including gross human error on my part.


Broad v. Local Aspect:  A slope's surroundings are as important as the aspect of an individual point.  Consider the screenshot below; NE and E aspects are shaded orange and the rest yellow.  If a forecast called for wind-loading of E aspects, it might be tempting to consider the yellow areas unaffected, but in reality the entire slope is on the east side of a major ridge.  Local variations in aspect are likely to be less important than the overall orientation of the slope, but CalTopo will not account for this.

Despite local north and south aspects, this entire slope is in the east shadow of a ridgeline



Runout Zones: At best, this shading will only help you determine avalanche starting zones.  Runout zones may extend well beyond the shading onto gentler terrain.

Despite being unshaded, the creek in this image is not a safe travel zone.

Data Limitations: Elevation data is provided by the USGS at approximately 30' intervals, but the original data may have been sampled more coarsely and then interpolated.  Slope and aspect calculations can only be as good as the source data, and may be off.

Snow Changes the Slope Angle:  If snow accumulates unevenly (like on the leeward side of a ridge), the slope angle on bare ground may not match the observed slope angle on top of the snowpack.  CalTopo's slope shading does not account for this effect.


Thursday, January 12, 2012

CalTopo's Current Map Layers

CalTopo map coverage for the contiguous (i.e. lower 48) United States is almost done.  Given that the map layers are about to be refreshed, this seems like a good opportunity to write about what is currently available, and where the data comes from.

First and foremost, USGS topo maps.  CalTopo's usgs layer is composed entirely of 7.5'x7.5' quads tiled together - somewhere in the neighborhood of 120,000 of them, although I don't know the exact number.  The USGS has 7.5' maps available from three different projects:
  • Digital Raster Graphics.  Although all their map scans are technically DRGs, the USGS uses this to refer to 250ppi maps scanned for Microsoft's Terraserver project.  These have the advantage of normalized colors (same shade of green everywhere) only occupying 238GB of drive space.  Downside is the low scan quality, which is evident in Terraserver's map tiles.  You can't get this online anywhere; you have to ship a drive to the USGS.
  • Historic Topographic Maps.  The USGS has been re-scanning every map they can find (not every map ever made, although coverage is quite good) at 600ppi and making the result available as georeferenced PDFs.  The scan quality is amazing, but the colors and even the intensity/darkness vary heavily from scan to scan.  These are called historical topographic maps, which is a bit confusing at first as it includes maps printed as recently as 10 years go.  More Information.
  • US Topo Maps.  The USGS has replaced their traditional printed maps with a product called "US Topo", the successor to their "Digital Map -  Beta" line.  You can still order printed copies of US Topos, but the source map is built electronically by compositing shapes and place names on top of vegetation coloring and aerial imagery.  These are available as multi-layer PDFs; you can pick individual layers with the right tool, but many PDF viewers show you all of the layers composited together (with aerial imagery covering up the vegetation layer).  Download US Topos.
CalTopo's topo layer is built from historic topographic maps, with US Topos filling in where historic topos were unavailable or improperly georeferenced.  The following states have not been scanned yet and are based off DRGs: WY, SD, LA, MS, FL, RI, MA, VT, NH.  California topos are based on a separate scanning project from Cal-Atlas.

The US Forest Service produces its own set of 7.5' quads with updated road and trail information, typically without vegetation shading; these are called Primary Base Series maps.  Those of you familiar with MyTopo's recently-restricted map layer may recognize that MyTopo appears to have used these in place of USGS quads where available, with green shading added on top.  Although CalTopo only has coverage for California, expansion to other western states is on the roadmap.

The USGS also supplies 1/3 arc-second elevation data (approx. 30ft) as part of the National Elevation Dataset, and this is used to produce the Shaded Relief and 40 foot contour layers.  Like the DRGs, this is only available if you ship a drive to the USGS.  Relief shading was done with GDAL, specifically the gdaldem utility at 3x vertical exaggeration, and then tiled.  The 40' contour layer was made using gdal_contour to produce contour vectors from the NED and mapnik to render those vectors into images. Both layers required a healthy dose of judgement calls; I tried to make decisions that would render best on top of a wide variety of backrounds, but the topo lines may be too thin on some layers and too thick on others.

The NAIP Aerial layer was built from the USDA's National Agriculture Imagery Program.  Google has higher-resolution imagery, but it seems to always have ugly seams right where you don't want them, and they photographed much of the Sierra when it was covered in snow, which makes the imagery useless.  I also wanted a cacheable version that we could put on a drive and take with us for SAR.  The existing layer was built from the .SID files on Cal-Atlas, but I plan to expand coverage using the image layer from US Topos at some point in the future.

The 1900 layer was built from scanned maps available on several sites linked to from http://www.sdc.ucsb.edu/holdings/caltopo.html.  All maps are from 1885-1915.  Datums were still in shift at this point so I had to make some educated guesses about the map's datum based on publication date; the result is that some maps are offset a bit.  The linked site has much better historical map coverage than the USGS, but I plan to expand coverage beyond California using the USGS's historical topographic map collection.

The Visitor map layer was pieced together from digital copies of USFS, NPS and BLM visitor maps.  Coverage is USFS/NPS in California, and national parks/national monuments in the western US.  I'd like to grow this layer in the Western US, especially the southwest, but it's painstaking work.  Each map requires georeferencing 3-4 pixel locations against real-world coordinates, and when two maps overlap, you need some surgical use of the eraser in Gimp or Photoshop.  

The OpenCycleMap layer is provided courtesy of the OpenCycleMap project; I have nothing to do with it.  It's the best vector-based climbing/hiking/biking map I've seen; it rocks in some locations, while trail coverage is sparse in others.

Terraserver images are provided by terraserver.com.  They're lower-quality and slower than the CalTopo layer, but cover Alaska.

MODIS maps are provided by the USFS Active Fire Mapping Program.  They're a little slow to load.