Out, d*mn spot!

The NOAA weather radar images from the National Weather Service have a reflectivity key on the side. To get rid of the ground clutter & other low intensity noise, just remove the colors from the image corresponding to returns below a certain threshold dBZ.  It sounds like it ought to be simple, but they threw a small monkey wrench into things.

Transparent NOAA radar composite without noise removalnoaa-dbz-scaleHere are a copy of the key clipped from the large composite map along with a part of the transparent radar-only composite on a checkered background. If everything at or below 5 or 10 dBZ is removed, most of the noise goes away, but the interesting weather remains. However, the composite national image does not consist of the pure colors shown in the key. Areas of the “same” color consist of a smattering of pixels that have almost the same value. It’s easy for your eye to see them as identical, but not quite as simple for a computer when comparing bits.

I entertained the idea of looking at the colors in terms of hue, saturation, and lightness instead of red, green blue. All pixels above a certain brightness and/or saturation could be removed. Those would correspond to white and grey areas, and to the light blue/cyan areas (below 15 dBZ) if the parameters are chosen carefully. Some initial tests looked promising, but then I discovered that the graphics tools I use (ImageMagick and GraphicsMagick) have a method that seems custom-made for this purpose — a method to turn transparent all pixels in an image of a given color, with a fuzz factor. The fuzz factor allows for matching all pixels close to the given color, and that’s just what is needed.

That’s enough of the bit-twiddling details — see the library documentation and my utility’s source code for the nitty-gritty. Lets look at the results.

radartrans2

At the right is the same radar-only transparent image from above with all returns of 10 dBZ or less removed, which corresponds to all shades of grey and the lightest two shades of cyan. The interesting weather remains.

And below are the two versions of the radar images superimposed on a Blue Marble image of the contiguous US. There’s no contest as to which looks better  (the right one).usradar-20090906-2059usradar-20090906-2059_dn

But am I throwing away too much?  Let’s compare  it to a commercial weather site image that has had the noise already removed.  I like Weather Underground for their forecasts as well as for their radar images and interfaces.  I’d bet dollars to doughnuts, though, that they don’t have their own nationwide network of doppler weather radar. They and all the others must be reworking the NOAA images that we in the U.S. have already paid for, adding value with their own reprocessing and presentation. 2xradarb3To the far right is a Weather Underground radar image from just about the same time as all the other images here.

usradar-crop-erTo the left of that is a section of the NOAA radar composite that roughly corresponds to the same area, overlaid on the Blue Marble, and prior to projection and rescaling.  The radar information content in both is very close. In fact, I’d say that Weather Underground has been more aggressive than I, and I’d probably be safe tossing the dark blue as well (15 dBZ).

Fortunately I’ve archived most of the original transparent NOAA composites. I’m reprocessing those now and will generate a new animation without those pulsating white blotches by and by.

Update 8-Sept-09: The new video has been processed and is embedded in the May 2009 derecho series article.  That article contains a link to the original video if you’d like to compare them.

Code

You can find the Perl script here. It will remove the low-dBZ reflections from the images found on the NWS website.  All the work to make every pixel of a single color transparent is done with a single call to the packaged method, which utilizes the GraphicsMagick library, so using Perl should be almost as fast as using C or another compiled language. The code simply loops through a list of colors to remove, and it writes the resulting image under another filename. You do have to use a format that supports transparencey (GIF or PNG, e.g.) to get the intended result.

Note that the version of ImageMagick or GraphcisMagic being used will affect whether this works as described.  After a lot of dependency hassles I gave up and finally upgraded my workhorse number cruncher from an older Fedore Core distribution to a recent Kubuntu version.  The packages for the older Fedora release  do not do what is needed here, and it was past time to upgrade anyway.  (I moved from Fedora to Kubuntu, as I have on other systems, to try to avoid these types of problems, benefiting from the longer lifecycle of Ubuntu’s long-term support versions. But don’t try to drag me into Fedora vs. Ubuntu vs. anything else wars!)

Buried in all the links in the text above is a link to the page with the radar images.  To make it easier to find, here’s the URL: http://radar.weather.gov/Conus/RadarImg/.  The world files for the various coverages are included, too.

8 Responses to “Noise removal from NOAA weather radar”

  1. egb13 says:

    @Boris, it doesn’t look like the link I put in my previous comment works? Here’s the image: egb13.net/content/wp-uploads/Radar_Zoom_colordiff.png

    One set of gray pixels have value #e3e3e3, and another set, visually indistinguishable, #e1e1e1. No use arguing about that — it’s in the data. So the question remains, what is the source that I carry on about vs. what you’re using. Could it be possible that NOAA has modified the generation of those images so that they’re no longer speckled?

    I’m out of town away from my systems that deal w/ this data. I’ll look into it further in a few days.

    Thanks for the interest and for the info!

  2. Boris says:

    Again, I need to insist that this is incorrect;

    “However, the composite national image does not consist of the pure colors shown in the key.”

    The image “latest_radaronly.gif” from the link you provide above does indeed consist of the “pure” colors shown in the key! I just downloaded it and dumped only the values encountered at least once in the image and their corresponding count;

    Value 0, encountered 4860339 times.
    Value 1, encountered 526 times.
    Value 2, encountered 1708 times.
    Value 3, encountered 5961 times.
    Value 4, encountered 19058 times.
    Value 5, encountered 49734 times.
    Value 6, encountered 119175 times.
    Value 7, encountered 220638 times.
    Value 8, encountered 65264 times.
    Value 9, encountered 43950 times.
    Value 10, encountered 28007 times.
    Value 11, encountered 15060 times.
    Value 12, encountered 6250 times.
    Value 13, encountered 3115 times.
    Value 14, encountered 740 times.
    Value 15, encountered 301 times.
    Value 16, encountered 106 times.
    Value 18, encountered 51 times.
    Value 19, encountered 16 times.
    Value 21, encountered 1 times.

    It is clear that there isn’t one pixel with value greater than 21. The key contains 22 colors from 0 to 21.

    If you would like I can send you my code (C#) so you can verify for yourself. I have been working with this image for sometime now and have not encountered what you describe above. Also I have a noise removal filter which does trim the lowest dBZs but also does heuristic filtering based on patches (isolated islands of pixels). I can share that as well.

    Best regards,
    Boris

  3. egb13 says:

    @Boris,

    Unfortunately the images I’m using are not as you describe — they’re speckled to a degree undiscernible to the eye but speckled nonetheless. I’ve uploaded an annotated example of that here: http://www.egb13.net/?attachment_id=682

    I’ll have to dig, but I don’t think anything I’m doing is introducing that speckling. I’ll follow up a second time if I find that’s the case. But given that others have found a need for my noise removal code, I don’t think the problem is here.

  4. Boris says:

    I don’t think this is correct;

    “However, the composite national image does not consist of the pure colors shown in the key. Areas of the “same” color consist of a smattering of pixels that have almost the same value. It’s easy for your eye to see them as identical, but not quite as simple for a computer when comparing bits.”

    The image at (radar.weather.gov/ridge/Conus/RadarImg/latest_radaronly.gif) is an 8 bit image and after I dump the palette I see the exact same ramp of colors as the key.
    What am I missing?

    Thanks!

  5. egb13 says:

    @John B,

    I just tried that example version and it didn’t work right — there must be some difference to what I have running on my batch system. :-/

    I hunted down the last version of the script that I’d done for someone who needed it to run on a Windows system. It works just fine on Linux, too. This version has been enhanced to allow for automatically uploading the modified image somewhere via FTP; run the command w/o any arguements and it will show you the options for the command line.

    I just tried the latest noaadenoise.pl script on my Ubuntu laptop and it worked perfectly against the file at http://radar.weather.gov/Conus/RadarImg/latest_radaronly.gif. Let me know if you continue to have issues with it, with more details of your environment and what image you’re using as input.

    To get it to work on Windows I believe I had to use ActiveState Perl with the Image::Magick module installed with the ActiveState tool for module management (don’t recall the name of that tool; I’m a *nix geek, mostly).

    The new script is in the same location: http://egb13.net/code/noaadenoise.pl.txt

  6. egb13 says:

    @John B: The example script runs every 10 minutes on one of my Linux systems creating the images for the animation at http://egb13.net/weather/radar/.

    There may be issues with whether you’re using ImageMagick or GraphicsMagick modules, whether you’re on Windows, and if on Windows, where you got the *Magick module for your version of Perl. I worked on a Windows version with someone else a while back, and it took some diddling to get it to work right. I don’t think I have the changes around here any more, though. :’( But if you tell me what you’re using I’ll see if I can help.

  7. John B says:

    Does your Perl script still work as written in your example?

    I have been trying to get this to work on the Ridge Radar images, having changed the colors to match the dBZ scale used with them, and none of the colors are being changed in my output image.

    Just curious if something changed since this was written?

    Thanks!

  8. Good stuff, I anticipate reading even more.

Leave a Reply

Your comment will be silently rejected if it contains a link. It's fine if you can mention a relevant site in a way that a human will be able to recognize it, but not as a full-blown link. Do not enter anything containing "http:". Sorry, this is necessary to fight link spam.