Fast downloading of GRIB2 Files from the NCEP NOMADS Server
(Partial HTTP Transfers)

Introduction

Using this procedure, you can download just the data you need from a remote GRIB2 file.

Note that in the examples below, the convention for the index files uses the ".idx" suffix, used on both the WOC ftpprd.ncep.noaa.gov server and the nomads.ncep.noaa.gov server.

These instructions were adapted for this site from Wesley Ebisuzaki's Fast Downloading of GRIB files website.  Please visit this site for more details.

Details

The http protocol allows "random access" reading of GRIB files.  This procedure requires an index file and an http program that supports random access.   The wgrib inventory is used for the index file, and cURL is used for the random-access http program.   Both are freely available, widely used, work on many platforms and are easily scripted/automated/put into a cronjob.  Two perl scripts are also required, get_inf.pl and get_grib.pl,  which are downloadable from the NCEP CPC website (see "Requirements" below).

The basic format of the quick download is:

   get_inv.pl INV_URL | grep FIELDS | get_grib.pl GRIB_URL OUTPUT  

• INV_URL is the URL of a wgrib inventory, for example:
  http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2008120200/gfs.t00z.pgrbf12.grib2.idx
• FIELDS is string that selects the desired fields (wgrib compatible), for example:
  ":HGT:500 mb:"
• GRIB_URL is the URL of the grib file, for example:
  http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2008120200/gfs.t00z.pgrbf12.grib2
• OUTPUT is the name of the for the downloaded grib file
  

The "get_inv.pl INV_URL" downloads the wgrib inventory off the net and adds a range field.

The "grep FIELDS" uses the grep command to select desired fields from the inventory. Use of the "grep FIELDS" is similar to the procedure used when using wgrib to extract fields.

The "get_grib.pl GRIB_URL OUTPUT" uses the filtered inventory to select the fields from GRIB_URL to download. The selected fields are saved in OUTPUT.  

See the wgrib home page for more information and tricks on using grep and egrep.

Examples

This example downloads the the 500 mb height from the 12 hour (f12) from the 00Z (t00z) GFS fcst from the NCEP NOMAD server:

INV_URL="http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20081202100gfs.t00z.pgrbf12.grib2.idx"
GRIB_URL="http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2008120200/gfs.t00z.pgrbf12.grib2"
get_inv.pl $INV_URL | grep ":HGT:500 mb:" | get_grib.pl $GRIB_URL out.grb

This example is similar to the above example except it downloads both the 500 mb height and the 1000 mb temperature:

INV_URL="http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20081202100gfs.t00z.pgrbf12.grib2.idx"
GRIB_URL="http://nomad.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2008120200/gfs.t00z.pgrbf12.grib2"
get_inv.pl $INV_URL| grep "(:HGT:500 mb:|:TMP:1000 mb:)" | get_grib.pl  $GRIB_URL out.grb


If you want to download multiple fields, for example, precipitation and 2 meter temperature, you can type,  

     URL="http://nomads.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2006070312/gfs.t12z.pgrb2f00.grib2"
     get_inv.pl $URL.idx | egrep ':(PRATE|TMP:2 m above gnd):' | get_grib.pl $URL out

The above code will put the precipiation and 2-m temp in the file "out". Egrep understands regular expressions which is a very powerful feature.

If you are doing multiple downloads from the same file, you can save time by keeping a local copy of the inventory. For example:
     URL="http://nomads.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2006070312/gfs.t12z.pgrb2f00.grib2"
     get_inv.pl $URL.idx > my_inv
     grep ":UGRD:" < my_inv | get_grib.pl $URL ugrd.grb
     grep ":VGRD:" < my_inv | get_grib.pl $URL vgrd.grb
     grep ":TMP:" < my_inv | get_grib.pl $URL tmp.grb
 
The above code saves two extra downloads of the inventory. 

Requirements

1. perl
2. grep
3. cURL
4. grib files and their wgrib inventory on an http server
5. get_inv.pl
6. get_grib.pl

Configuration (UNIX/Linux)

The first two lines of get_inv.pl and get_grib.pl need to be modified. The first line should point to your perl interpreter. The second line needs to point to the location of curl if it is not on your path.

Usage: Windows

There have been some reports that the perl scripts didn't work on Windows machines. Please see Wesley Ebisuzaki's Fast Downloading of GRIB files page for more information and solutions to Windows access problems.