There are a couple restrictions on the type of image you can use. First, it has to be hosted online somewhere - web server, flikr account, whatever. CalTopo won't upload and host the image for you. Second, it needs to be drawn to scale. CalTopo will scale and rotate an image, but it won't do any warping.
Custom layers are shown on the left nav bar below markers and shapes. Click New Layer to get started.
First, supply the image URL and click Set.
The image should appear on your screen, with the bottom left and upper right corners aligned to the corresponding corners of the map viewer. You can mark the two image points, and their corresponding real-world coordinates, by right-clicking on the map background and selecting Georeference.
Use the opacity slider on the custom layer dialog to hide the image and show the underlying map. Once you've marked both point/coordinate pairs, you can fine tune the results by continuing to click on the map or through the dialog. When done, click save.
Once the layer has been created, it appears in the overlay dropdown along with all the other layers. You can't use the custom layer as a base layer.
If you're working on named map, the custom layer will be saved along with that map. If you're just using the regular map browser, the custom layer will be added on to the URL hash string, so that you can bookmark it or send a link to other people without saving anything to CalTopo. As an example, here's the Huddart County Park map with shaded relief and contours. Click here to see it in person.