An Introduction to Meteorological Data Management
and the WXP Software Package in SFSU's
Weather Graphics and Simulation Laboratory (WGSL)

Table of Contents
  1. Meteorological Data
  2. Local Data Manager (LDM) Programs
  3. Weather Processor (WXP) Programs
  4. Questions about LDM and WXP Programs and about DDS Data
  5. Running WXP Programs
  6. Telling WXP Programs Exactly What You Want Them to Do


I. Meteorological Data

At SFSU we receive seven streams of weather data that come ultimately from the National Weather Service (NWS), the National Centers for Environmental Prediction (NCEP), and other sources. These data streams arrive via the internet and include:
  1. Domestic Data Service Plus (DDS+) data, consisting of surface, synoptic, radiosonde, buoy and radar observations, and frontal analyses, plus NWS forecasts and discussions, weather watches and warnings, significant weather observations, and a forecast-model analysis product called "Model Output Statistics" (MOS);
  2. International Data Service (IDS), consisting of weather observations from abroad;
  3. High Resolution Service (HRS), consisting of numerical forecast-model output;
  4. Lightning strike data from the National Lightning Detection Network (NLDN);
  5. MCIDAS satellite images, of several different types;
  6. NEXRAD doppler radar data; and
  7. profiler data (vertical radar profiles of wind and temperature)

DDS and IDS data have to be decoded after one of our computers ("enso") receives and stores them. Figure 1 shows how the DDS+ and IDS data, for example, flow through a series of programs that decode and ultimately display the data either graphically or as text. Sections II and III below briefly describe what happens.

II. Local Data Manager (LDM) Programs

We have programs in the Local Data Management (LDM)1 software package running continuously on enso and norte. These programs request and monitor the streams of meteorological data arriving via the internet, decide which data to save, and save or reject them, following instructions that we can specify2.

LDM programs save data in the same format in which they arrive. These data may consist of text (e.g., written NWS forecasts); coded observations (e.g., surface observations, radiosonde observations, radar observations, etc.); or numbers represented in non-decimal formats (e.g., binary data stored in binary files, such as the HRS, NLDN, MCIDAS, NEXRAD, and profiler data). These data formats are usually hard for people to read, and in some cases analysis programs and graphics programs can't read them either, so some types of data have to be converted or decoded to become useful.

To make raw data useful, UCAR's Unidata program provides and supports several software packages that parse, convert/decode, analyze or graphically display weather data. These packages include:

All of these packages are occasionally upgraded. At SFSU, as of Fall 2004, we use LDM version 6.1 and WXP version 5.32 or so.

III. Weather Processor (WXP) Programs

Parsers and Decoders

Raw text data can be made easier for humans to read by parsing the data into smaller files containing selected information. Programs in the WXP software package that do this are called parsers.

Examples include:

WXP Program Purpose
griblook parsing computer model output data
fo_parse parsing Model Output Statistics (MOS data)
parse parsing text data
sa_parse parsing standard atmospheric observation (SAO data)
ua_parse parsing radiosonde data
sacvt decoding SAO data
smcvt decoding synoptic data and ship/buoy data
radcvt for decoding radar data
uacvt for decoding radiosonde data

Observed data (from the DDS and IDS data streams) can be made easier for graphics and analysis programs to read by converting/decoding the data into one of several formats, including netCDF ("network common data format", a binary format) and specially formatted text files unique to WXP. WXP programs that convert raw DDS and IDS data to binary netCDF or specially formatted text files are called decoders.

Plotting Programs

Once converted into netCDF or specially formatted text, other WXP programs can read the observed data and plot it directly on the computer screen or send it to a laser printer. These are called plotting programs. Plotted data displayed on the screen can take the form of numbers, station models, radar echoes, etc. Other plotting programs can plot some kinds of raw data—such as HRS or NEXRAD data—directly, without converting/decoding it first. Examples include:

WXP Program Purpose
fouswx plotting Model Output Statistics (MOS)
mapplt plotting geographic data, such as maps
rad plotting radar data
sfcwx plotting surface observations
uacalplt plotting radiosonde soundings
upairwx plotting radiosonde observations

Interpolating and Contouring Programs

Observed data are distributed irregularly in space because weather stations and radiosonde launch sites are. However, for some purposes we can more easily analyze or display the data if the data are interpolated to a regular grid of points evenly distributed in space. WXP programs that draw contour analyses, for example, first interpolate data to a regular grid, and there are other programs that display wind vectors on a grid. These are examples of interpolating and contouring programs. Examples include:

WXP Program Purpose
focalc contouring Model Output Statistics (MOS)
grbcalc contouring computer model output
sfccalc contouring surface observations
uacalc contouring radiosonde observations
xsat contouring satellite image data

Interpolating and contouring programs all display the gridded data on the computer screen or send them to a laser printer, but first they create files containing the gridded data. These files are called grid files.

Calculation Programs

Several WXP programs read netCDF data or specially formatted text files, perform calculations with it and print the numerical results. These are calculation programs, which produce no graphics or additional files. Examples include:

WXP Program Purpose
heat calculates heat index
moist calculates various moisture variables
stdatms relates pressure, height, and temperature in a standard atmosphere
suncalc compute sunrise, sunset, twilight
unit convert units
wchill calculate windchill factors

General Purpose and Other Programs

WXP includes a number of other programs that allow you to overlay or animate multiple images; use other WXP programs more easily; access WXP database files containing names, latitude/longitude locations, and elevations of weather stations, as well as other non-weather information; perform mathematical functions on raw or gridded data; etc. Only the last-mentioned programs operate on weather data, and most of them do not create additional files.

For information about any WXP program, see the WXP Users Guide (http://wxp.unisys.com/wxp/wxp5/Users/index.php) and the WXP Program Reference (http://wxp.unisys.com/wxp/wxp5/Program/index.php).

IV. Questions about LDM and WXP Programs and about DDS Data

OK, so we've got a bunch weather data and programs that ingest, sort, save, parse, decode, grid, analyze or display the data. BUT:

  1. Where are LDM, WXP, and other programs kept, and what are the WXP programs called?

  2. How do we run LDM, WXP, and other programs?

  3. How are all the data files named, and where are they, including:
    1. files of unmodified (undecoded) data (including text, coded observations, and binary files);
    2. files of parsed text data, and files of decoded observations;
    3. grid files (in netCDF format).
    4. "raw" files (another special WXP text format).

In partial answer to question (1) above, the LDM software is kept in directories under the home directory of an account called "ldm". WXP programs are kept under the home directory of an account called "wxp". In partial answer to question (3), weather data are stored in subdirectories of directory "/data". Using UNIX commands on a workstation, you can explore the directory hierarchy to see more precisely where the files containing data and programs are stored and how they are named.

The answer to question (2) ("How do we run the programs?") depends on the program:


V. Running WXP Programs

There are several options for running WXP programs, described below.

The Menu Approach

The easiest way to run WXP programs is first to run the interactive WXP program called "wxp", which presents you with menus from which you can make selections that in turn tell the computer to run WXP decoding, plotting, contouring or other programs for you:

norte_wxpuser% wxp
WXP: The Weather Processor - version 5.32-Solaris-X11

    Main Menu

 1: Parsing Programs
 2: Plotting Programs
 3: Contouring Programs
 4: Meteorological Calculations
---------------------------
 0: Return to previous menu
-1: Exit WXP shell

WXP-main> 

This approach is very easy because it doesn't require you to know anything at all about other WXP programs or how they work. However, it's sometimes tedious to work your way through all the menus. It's also not very flexible—there are many things that WXP programs can do that you can't do using only the menus presented by the wxp program. Hence, for some purposes you should consider use of wxp just a "training wheels" approach to using WXP programs.


The Command Line Approach

The UNIX operating system always prompts you when it's ready to accept a command, and we have control over the appearance of the prompt. For someone logged onto account "wxpuser" on workstation "norte", we've designed the prompt to look something like this:

     norte_wxpuser%

You would respond to this prompt by typing a command on the same line. This line is called the command line. (In examples of WXP commands below, the prompt is left out for convenience.)

You can run any single WXP program by typing the program name, followed by information needed by the program to do what you want, directly on the command line. For example,

     sfcwx  -current  -region us  -variable all  -device d

will run the WXP program called sfcwx, which plots surface weather data. In particular, it will plot data from the current hour ("-current") for the whole U.S. ("-region us") in the compact form of station models ("-variable all"), and display it graphically on the computer screen, which is also called the display ("-device d").

For convenience, the same command above can be shortened to:

      sfcwx  -cu  -reg us  -var all  -dev d
or
     sfcwx  -cu  -re=us  -var=all  -dev=d


and sfcwx will still understand what you're telling it to do.

If you don't provide on the command line a piece of information needed by the program, the program will either (a) choose default values for you; or else (b) prompt you for the information, often by presenting you with a "menu" of options from among which you select one.

The Script Approach

Two very powerful things that you might want to do with weather data include (a) overlaying two or more "fields" of information, such as wind vectors and contours of pressure or geopotential height, to see more clearly how they relate to each other; and (b) animating sequences of images in "loops". It is possible to generate loops of images using single WXP programs, but it's difficult to overlay fields using just one program. Instead, you usually have to run a sequence of WXP programs (possibly interspersed with UNIX commands), coordinated to achieve what you want. Sometimes it's better to generate animated loops this way, too.

Learning to use WXP programs this way is not easy, but there is a way to "package" sequences of WXP programs, UNIX commands and other programs into a single program, called a script, to create overlays and loops. At SFSU, Dr. Dempsey has written a variety of scripts to make creating overlays and loops relatively easy. You "run" these scripts on the command line, much as you would run a single WXP program. For example:

     bayarea -cu 1

will run a script called bayarea, which overlays surface weather data (in the form of station models) on top of radar echoes, for a part of central and northern California that includes the S.F. Bay area. In this case in particular, data recorded one hour before the current standard observing time are displayed on the computer screen ("-cu 1").

Another example:

     xsatloop ir -nu 24

will run a script called xsatloop, which creates an animated loop of satellite images of a type that you specify for a length of time that you specify. In this case in particular, each image consists of an infrared satellite image from the GOES-West satellite ("ir"), which covers the eastern Pacific Ocean and western U.S., and there will be 24 images looped ("-nu 24") at intervals of one hour.

Lists of the WXP scripts that you can run are posted on the bulletin board in TH 607. You can find a similar list, with links to more information about how to run the scripts, at http://funnel.sfsu.edu/wgsl.

The Interactive Approach

The script approach makes creating overlays and loops relatively easy, but you have a limited choice of overlays and loops that you can create, and once you've created them you have no control over them.

However, it is possible to run from the command line, in sequence, the same WXP programs that appear in a script (if you know what they are!). This is more tedious and requires much more sophisticated understanding of how certain WXP programs work, but it does allow you to use certain WXP programs from the command line to delete images from or add them to a loop; overlay additional fields of data, remove fields already overlain; add or remove labels and titles; and so on.

In short, by typing individual WXP commands on the command line, you have much more control over the result that appears on the screen. You can "interact" with the computer to create your own overlays and loops and to change them after you've created them. But it takes a lot of practice to master this approach!

VI. Telling WXP Programs Exactly What You Want Them to Do

Parameters and Arguments

You have to give most WXP programs information about what precisely you want them to do. This is true even if you just use the wxp menu program to run WXP programs.

For example, if you want to plot surface data, you could run the sfcwx program from the command line, but at a minimum you have to tell sfcwx the particular variable that you want to plot (station model, temperature, pressure, or whatever); the region of the world you want; the observation time you want; and where to send the output (to the "display" or to a laser printer). In addition, if you want, you can tell sfcwx what size, color, font type, and location of labels and titles to use on the plot; whether you want all stations plotted or just some; the location of the file containing the decoded data; the location of files containing latitudes and longitudes of weather stations (so the program knows where on the plot to place the observations); the location of a file containing the map background; and so on. In fact, for the sfcwx program to work, it needs to know about three-dozen different things!

Pieces of information that programs need to work we call parameters or arguments (or resources, as WXP calls them). The values of various parameters needed by sfcwx and other WXP programs have to be specified for them to work.

Default Values for Parameters

It would not be practical for us to specify the values of all three-dozen parameters each time we wanted to run sfcwx. Fortunately, many of the parameters have values that are very common or standard and are not likely to change from one execution of the program to another. Such parameters can be given default values that the program will use unless you tell it otherwise.

We specify many of the default parameter values for WXP programs in a WXP "resource file" usually called "Wxp.res". When you run a WXP program, the program automatically checks the resource file for the default values of many parameters, and prompts you (often by presenting a menu of options) for the values of parameters for which there is no default and that you haven't otherwise specified.

We keep the main resource file, Wxp.res, in directory /usr/local/unidata/wxp/etc, but you can set up your own local version (with the same name) somewhere else, and tell WXP that yours will override the main one.

Specifying Parameter Values

If you don't specify the value of a parameter for which there is no default value, then WXP programs will prompt you for a value, often by presenting you with a menu of options from which to chose. For example, when you run the program "wxp" and thereby take the "menu" approach to running WXP programs, you'll always get menus of options for some parameter values. Your selections from the first two menus presented by wxp always determine which WXP program will run, but subsequent menus present options for parameter values required by the program you've selected. For example:

norte_wxpuser% wxp

WXP: The Weather Processor - version 5.32-Solaris-X11

    Main Menu

 1: Parsing Programs
 2: Plotting Programs
 3: Contouring Programs
 4: Meteorological Calculations
---------------------------
 0: Return to previous menu
-1: Exit WXP shell

WXP-main> 2

    Plot Data Menu

 1: Plot Surface Data
 2: Plot Ship Buoy Data
 3: Plot Surface Meteograms
 4: Plot Upper Air Data
 5: Plot Soundings
 6: Plot ETA Model Soundings
 7: Plot RCM Radar Data
 8: Plot USRad Data
 9: Plot NOWRad Data
10: Plot Lightning Data
11: Plot MOS Data
12: Plot MOS Meteograms
13: Plot Maps
---------------------------
 0: Return to previous menu
-1: Exit WXP shell

WXP-plot> 4
Running: upairwx 
      UPPER AIR DATA PLOTTING (Ver 5.32-Solaris-X11)

List of available files:
/data/decoded/04092600_upa.asc  /data/decoded/04092612_upa.asc
/data/decoded/04092700_upa.asc  /data/decoded/04092712_upa.asc
/data/decoded/04092800_upa.asc  /data/decoded/04092812_upa.asc
/data/decoded/04092900_upa.asc  /data/decoded/04092912_upa.asc
/data/decoded/04093000_upa.asc  /data/decoded/04093012_upa.asc
/data/decoded/04100100_upa.asc  /data/decoded/04100112_upa.asc
/data/decoded/04100200_upa.asc  /data/decoded/04100212_upa.asc
/data/decoded/04100300_upa.asc  /data/decoded/04100312_upa.asc
/data/decoded/04100400_upa.asc  /data/decoded/04100412_upa.asc

Enter the upper air filename: 


and so on.

A drawback of using the "menu" approach to running WXP programs is that you can't change the default value of any parameter that actually has a default value.

However, you can to override the default values of parameters (as well as specify the values of parameters for which there is no default) using the "command line" approach. The general form for specifying parameter values on the command line is:

     program_name  -par_name1 val_par1  -par_name2 val_par2  ....
or
     program_name  -par_name1=val_par1  -par_name2=val_par2  ....

where program_name is the name of a WXP program. The other stuff on the command line perform two functions:

  1. Identify the parameters. In the general form above, par_name1, par_name2, etc. represent the full or abbreviated names of parameters required by WXP program program_name. Each parameter name has a dash (" -") attached to it, which tells the WXP program that what immediately follows is the name of a parameter (as opposed, say, to the value of a parameter). Each dash must be preceded by a space.

    Some examples of parameter names include:
  1. Specify the value of each parameter. In the general form above, val_par1, val_par2, etc. represent the values assigned to the corresponding parameters par_name1, par_name2, etc. Each value should immediately follow the corresponding parameter name, with a space or "=" separating them.

    Examples:

The observation time can be an exception to the convention for specifying parameter values if you specify it in the form yymmddhh—it's an ordinary argument that doesn't need a parameter name preceding it. Like parameters, you can put it anywhere on the command line (as long as it doesn't come between a parameter and its value).

For example:

     sfcwx  03030700  -var slp  -re ep  -dev d
or
     sfcwx  -var slp  03030700  -re ep  -dev d 
or
     sfcwx  -var=slp  -re=ep  -dev=d  03030700 

All of three of these would plot the same thing—sea-level pressures at the location of each station in the Eastern Pacific region observed on March 7, 2003 at 00Z, plotted on the screen.

For a full summary and documentation of WXP parameters, see the WXP Resource Guide (http://wxp.unisys.com/wxp/wxp5/Resources/index.php).



    Table of Contents
  1. Meteorological Data
  2. Local Data Manager (LDM) Programs
  3. Weather Processor (WXP) Programs
  4. Questions about LDM and WXP Programs and about DDS Data
  5. Running WXP Programs
  6. Telling WXP Programs Exactly What You Want Them to Do

Footnotes
1. The LDM package is provided free by Unidata, a program run by UCAR (University Corporation for Atmospheric Research) in Boulder, Colorado to provide and support meteorological data-management and graphics software for universities.

2. The LDM programs follow instructions that we put in "configuration files" called "pqact.conf". They are located in directories ~ldm/etc and /usr/local/unidata/ldm/etc.