AVIATIONTOOLBOX: how I convert sectional maps to map chunks

For the past few weeks, I've been obsessed with manipulating the newly-
available (from the FAA) raster sectional images. This is all new to
me so I've just been banging out code trying to get something
reasonable. The code is *horrible* and I don't recommend it as
anything beyond an example of something CS graduates should know better
than to generate. However, I've been asked for details about it, so
here goes...

I subscribed to the Sectional Raster Aeronautical Chart Eastern Set
(SRACE) and Western Set (SRACW) from the FAA.
http://www.naco.faa.gov/index.asp?xm...ctional_Raster
There's also an Alaska Set (SRACA) but I wanted to get comfortable with
these before trying to justify the cost of it.

These DVDs are supposed to come every 28 days. I've only received one
set so far and it expired on November 27. Gotta call about that.

I uploaded the data to a server I got in Chicago primarily for this
purpose. I'm not sure how the layout will evolve, but for now the
data is here.
http://aviationtoolbox.org/raw_data/FAA_sectionals/

I cropped the cover image from the St. Louis chart so that I could
easily get a handle on which regions are where.
http://aviationtoolbox.org/raw_data/...al_regions.png
(I have copies of this marked up all over the house.)

In the FAA_sectionals/ directory there are East and West compilations
(zip files), and the data is available in the original (uncompressed)
format in the current/ directory. The images are fairly compressible
so if you want a bunch, get the compilations.

With compression you can easily fit everything on a single DVD. Seems
like a sales opportunity...

Now that the data is on the server, I announce that it's available and
the server is flooded with requests. I'm on a low bandwidth plan, so
I can only push about 6Mbps. For several days, that's exactly what
the machine did. Now it's fairly idle. You're welcome to peek at
what it's doing.
http://aviationtoolbox.org/server-status
(Note that there's apparently a bug in Apache2 - the request and idle
counts are always zero. Just look for the _W_aiting children.)

O.k., so now everyone has the GeoTIFFs. What happens? Well, lots of
people are already familiar with this data and have applications to
handle it and don't need anything more from me. Great!

I want to make the data more accessible to everyone to build their own
tools though. To me that means combining all of the maps and geting
rid of the non-map data (legends, etc.). That's what I've been doing
recently.

The first step was to decide how to determine where the map part of
the image is. I wrote some programs to use white space to find the
map data, but there are enough inconsistencies that I decided to hand-
code the lower left corner locations (longitude, latitude). I used my
automation to do this but then I checked it against the cover map. I
found that some corners couldn't be determined from that map, so I
made thumbnail images of all the corners and checked those.
http://aviationtoolbox.org/munge/corners
From this, I generated the list of corner locations.
http://aviationtoolbox.org/munge/sectional_corners

The next step is to mask off everything that's not part of the map
image. Given the lower left corners, it's not too bad. (I'm still
tweaking it though.) I create a mask that has zeros for everywhere
around the map image and then scales from 1 to 255 from each edge.
I'm tentatively calling this the "quality" of the pixel, because
things get complicated near the edges.

When I write the image, I convert all of the pixels from the 8-bit
palette to 24-bit RGB. Any pixel with a zero mask value becomes
white (255, 255, 255).
http://aviationtoolbox.org/munge/mas...quality_py.txt
http://aviationtoolbox.org/munge/masked/

Now that I have just the map image, I need to rotate (warp) the image
so that it lines up squarely with all of the others. To understand
why you need to do this, look at the map on the cover.
http://aviationtoolbox.org/raw_data/...al_regions.png
If you physically tried to align all of the printed sectional maps,
you'd end up with them bowing down toward the middle. That's too
complex for me to handle, so I picked a common reference sectional.

Everything is warped to the geotransform (slope) of Wichita North.
To the west, sectionals are rotated a bit clockwise, and to the east,
they're rotated counterclockwise. This warping takes a long time -
and I do it twice. The first warp is for the RGB image data. It
uses a cubic spline resampling algorithm. This returns a great-
looking image by weighting a large block of source pixels to
determine what each destination pixel will be.

The cubic spline resampling doesn't work well with a fourth band (the
mask), so I handle it on its own. I execute the same warp, but with
the much-simpler "nearest neighbor" method of determining the
destination pixel.
http://aviationtoolbox.org/munge/warped/

Note that the GDAL library doesn't include Python bindings for warp
operations. I had to resort to C++.
http://aviationtoolbox.org/munge/warp.cc
The warp operation also handles the *slight* difference of pixel sizes
of the various images. They're all close to 64 meters/pixel, but in
the warp I force them to be exactly that.

We almost have a continuous data set now. The last step is to combine
the warped images. They go together easily now that they're all
warped to the reference image. Except...there's overlap.

The overlap is why I originally set up the mask channel to have the
"quality" values. I was thinking that whenever there's overlap I'd
just pick the pixel with the highest quality (furthest from the edge
in its original image). This gives a nice sharp image, but it's
possible to lose data because some text could be *only* on the edges
of both images.
http://aviationtoolbox.org/tmp/chunk_furthest.jpg

I didn't feel good about that approach so I tried blending instead.
This sounded too difficult at first, but because GDAL uses Python's
Numeric extensions (arrays) for the image data, it was amazingly
straightforward. The blending algorithm multiplies each pixel's value
(for each band - RGB) by the pixel's weight. It then adds this to the
existing weighted pixel and then divides by the total weight (or one
if there's no good pixel there).

(Yes, if there's more than two images that overlap on an image (like
in the Denver/Wichita/Albuquerque/Dallas area), the subsequent images
are weighted more just because the earlier images have already been
averaged. There are ways around that, but I don't yet think it's
worth fixing.)

The result is a much more gradual transition between images.
http://aviationtoolbox.org/tmp/chunk-blended.jpg
Info on the edges can still be seen. Often the same information
appears twice. Sometimes it looks "jittery." More work to do on
that.

This blending all occurs as part of the "chunking" process. Because
TIFFs are limited to 4GB total size, I can't just throw everything
into one big uncompressed file. I could resort to using an ERDAS
Imagine (.img) file, but I decided smaller files would be easier to
manipulate for now.

Another constraint is that GDAL doesn't seem to like to create small
GeoTIFF files. The answer was to make the images 5000x5000 pixels.
(That's 320Km x 320Km.)

I "chunk" each masked image by moving across the masked image,
getting the chunk that corresponds to the current point, copying
everything in the masked image that fits into that chunk, and then
moving past the edge of the chunk in the masked image to repeat the
process with a new chunk. Each chunk is named by how far it is
from the Wichita reference point. The first chunk to the left is
"chunk_-1_0.tif" and the one above that is "chunk_-1_-1.tif".

So...that's it! It's amazing that I could spend so much energey on
something so simple, but it's been quite an education.

various notes...

I could not have accomplished this without GDAL.
http://www.remotesensing.org/gdal/
Writing this in anything but Python using a well-crafted library like
GDAL would have been unbearable. Frank Warmerdam gave me crucial
insight and example code.

Along the way I build a lot of JPEG and PNG files. These are usually
so that I can easily check on the process without loading a huge TIFF
file. The PNGs will show the mask/quality value as the transparency.
This means that the edges show up well and the middle portions of the
images disappear. That makes it easier to spot problems. Sometimes
I build tools to display these small images.
http://aviationtoolbox.org/munge/mosaic
These images are just for reference.

Each sectional (North and South sides) takes from 40 to 60 minutes to
process on this machine (2.4GHz Xeon). There are 37 sectionals. (I
don't deal with Hawaii now.) It takes awhile to try something new.
Right now I'm regenerating everything using additional masking
operations to trim the right and bottom edges. It won't finish for a
couple days - and that's if I don't change my mind and try something
different.

Everything could change. Don't count on these URLs remaining the
same. I'm still getting comfortable with this data.

There are lots of places where processing efficiency could be
improved greatly. I'm just trying to get something that works now.

It might be that the chunked data isn't what you want. It could be
better, for example, to use the warped data and make your application
decide which sectional to use on edges. The chunks are for ease of
use, but they're not perfect.

I prefer that comments/questions/extensions/etc. that might benefit
others go to the newsgroup(s). Please use "AVIATIONTOOLBOX:" in the
subject to help those who want to follow this and those who want to
skip it.

Please enjoy the data!

--kyler

Hm. I don't know if you noticed or not, but the terminal area charts
inside the current-east and current-west zip files have at least some
duplicates -- one I noticed in both was the DFW TAC. You may want to
fix this for future updates -- I know you probably won't change the
files now -- it might be a pain for some people who are currently d/ling
or resuming their d/l off your website -- but you can fix this for
future updates.

Matthew F. G.

"Matthew F. G." writes:

Hm. I don't know if you noticed or not, but the terminal area charts
inside the current-east and current-west zip files have at least some
duplicates

Yup, there are several.

I was trying to leave those collections just as I received them from
the FAA (duplicates and all).

I'm hoping to do a *much* better job at distributing the files once I
see how the next set (due last week) looks. I'm thinking that I'll
probably have an rsync interface at least.

--kyler