Or: Making Interactive Web Maps with USGS Topobathymetric Lidar Data
Hurricane Sandy slammed into New Jersey in 2012, causing widespread devastation. For the 2013 sailing season, no one knew exactly what to expect. The storm was so powerful that it changed the shape of some of the barrier islands that delineate Barnegat Bay—it cut one in half completely, temporarily creating a new inlet that allowed the ocean to flow in and out of the bay. While avoiding an island is fairly straightforward, many sailors were concerned that other, less visible features of the bottom of the bay had changed. Sailboats have relatively deep keels and Barnegat Bay is relatively shallow, which means that we run aground from time to time in a good season, when we’re reasonably sure that our charts are correct.
Jen and I attended a meeting in February 2013 at the Cedar Mar Yacht Club where we saw an interesting presentation from the US Geological Survey (USGS) that included something billed as “Bottom Analysis and LIDAR Radar.” Lidar (also written as LIDAR, LiDAR, or LADAR) sometimes gets jammed into acronyms like LIght Detection and Ranging or Laser Imaging, Detection, and Ranging, but it started out as a simple portmanteau of “laser” and “radar.” Just as radar uses radio waves or sonar uses sound waves, lidar bounces laser beams off of stuff to measure distances or detect objects. If you shoot enough lidar at the ground from, say, a moving plane, you can use it to collect extremely precise topographic data about the land. The interesting thing about the USGS presentation was that they’d flown “green laser” lidar over the bay. Most lidar uses infrared lasers, which can’t see through water. Lasers in the green portion of the spectrum, however, can penetrate water and provide an image of the bottom of a bay or river. I was pretty sure that this was the ticket to finding exactly how outdated our charts had become.
And then several years elapsed.
It turns out that using green lasers is more complicated than standard infrared lidar, and as I lamented in June of 2013, the data wasn’t released in time for it to calm a sailor’s nerves. We sailed anyway, and found that most of our local knowledge was still good, but I never quite gave up on the idea of doing something with that data.
So this winter, in 2017, I’ve been thinking about racing. Revolution is coming off her most successful racing campaign yet, and we’re always thinking about ways to improve. One of the things that keeps racing on a small course on the Delaware River interesting is the current…something I don’t have to consider when sailing Fortuitous on most of the bay. Getting a boost from a favorable current or sailing a longer course around an unfavorable current can be the difference between winning and losing on the river, but the data on exactly how that works is scarce. There are stream gauges that report the current in a handful of places, but the reality is that for any given section of river, the current isn’t even uniform across its span.
And that’s when I remembered the lidar: The USGS flew lidar over all of New Jersey’s waterways, including the Delaware. I just needed a way to show it to Sailor Steve to see if we could use it to better understand how the current might flow over the contours of the riverbed.
The data is now freely available for download, but comes in large, unwieldy files called GeoTIFFs, which are not very useful without specialized geographic software. I happen to have specialized geographic software, and a nearly unlimited patience for downloading bizarre nonsense off of the internet, but I wanted a way to share it on the web with normals, and winter isn’t winter without a quest.
This turned out to be more difficult than I expected in a lot ways, but easier in others.
When I first pulled the file up in QGIS (an open-source desktop GIS application), I thought I was most of the way there. My plan was to tile the image it so that I could serve it in bite-sized pieces over the web. The GeoTIFF that I downloaded is very detailed—each cell in the image represents 1 square meter—but that makes the file over 2GB. When trying to share it on the web, that’s way too big, but in reality, no one ever needs to see it all at once. When someone is zoomed out, they can’t see all the pixels anyway, so the image can be simplified. As someone zooms in, they need progressively more detail, but they only need to see progressively smaller pieces of the map at a time. To accommodate this, we tile the whole thing, by representing the map at every possible zoom level and cutting it into squares of 256 x 256 pixels, which are pretty quick to download.
To tile it, I used GDAL (the Geospatial Data Abstraction Library). It’s a software library that I ran from the command line to create the approximately 500,000 tiles required to show this area of the river from zoom levels 8-19. Doing this work up-front allows the map to be panned and zoomed quickly on the web, but creating half a million tiles is not an instantaneous process, it took a couple hours (I think…I didn’t watch it). The problem was that every time I ran it, my tiles were completely bonkers. The whole landscape was a seemingly random mishmash of gray squares.
Eventually, with the help of some coworkers who actually know how this stuff is supposed to work, I learned that I needed to do a little pre-processing to the GeoTIFF before it could be tiled. Although TIFs are a kind of image file, and GDAL was able to apply styling to it to make it look reasonable, this GeoTIFF wasn’t really set up to be an image. Instead of each cell containing a color value, it contained an altitude in meters.
The solution was to apply a hillshade. This is a raster processing operation that simulates illuminating the map in light and implying shadows given an azimuth and altitude of a virtual sun. This also looked interesting in QGIS, but after several rounds of trying different settings with the tiling operation (each taking a couple hours to run) my tiles were still all nonsense.
It turns out that there are two different hillshades in QGIS. I was trying to use the hillshade under Raster > Terrain Analysis > hillshade, but what I needed to do (obviously) was Raster > Analysis > DEM (Terrain Model). The GeoTIFF is a DEM—a digital elevation model—and the terrain modeling converts the raw data into an image that can be tiled.
After some tweaks, I threw it all into a web-accessible Amazon S3 bucket, and I had an interactive map of the bottom of the Delaware River.
Here it is:
Not sure if it will help us with our racing or not, but that’s how you get from Hurricane Sandy to a map of the Delaware riverbed. It’s been a long winter.