Compiling and Running WAX

Author:: Serge Monkewitz

	Contents
	A Simple Example Using the WAX template files Editing the template files Generating source code and compiling WAX WAX executable Usage Command Line Arguments Optional Arguments ASCII Table Files Retrieval column file Output column file Output column file Grouped apparition column file Type mapping between Informix and C

A Simple Example

The following section demonstrates how to produce a WAX executable in it's simplest form: an executable which does not use the plug-in interface, and does not compute any group attributes beyond the standard ones listed in the documentation for the WAX output tables.

Using the WAX template files

The first step in the process is to create a subdirectory in the WAX installation directory, wax_simple say :


% cd /my_wax_install
% mkdir wax_simple
% cd wax_simple

Next copy over the provided template files, located in the template subdirectory of the WAX installation :


% cp ../template/* .

The following files will now be present in the newly created example directory :

```
bands.txt
```
```
output.tbl
```
```
grpapp.tbl
```
```
retrieve.tbl
```
```
gen.sh
```

Editing the template files

The first file, bands.txt, contains a series of strictly increasing declinations corresponding to band boundaries, and segments the sky into separate, manageable processing regions for the WAX executable. The provided template file splits the sky into 483 bands, and is suitable for processing the 2MASS Working Point Source Database. For the purposes of this example, it will be left as is. Details on the format of this file are available in the band file format section.

The next file, output.tbl, specifies which columns (beyond the standard ones) will be generated via the plug-in interface for the group information table. Since none are being generated, the file (empty) can be left untouched. See the output column file section for information on the expected file format.

The file grpapp.tbl, specifies which columns (beyond the standard ones) will be generated via the plug-in interface for the grouped apparition table. This empty file does not need to be edited because the example does not use the plug-in interface. The grouped apparition column file section contains details on the expected file format.

Similary, retrieve.tbl specifies which columns (beyond the standard ones) will be retrieved from the apparition table. Because no non-standard group attributes are being computed, this file can also be left empty. The file format is documented here.

The last file, gen.sh is a shell script which provides a convenience wrapper around the srcgen helper application, and does need to be edited, even for this simple example. There are five environment variables set in the script which must be tailored for the example :

APP_TBL
TEST_DB
NAME_KEY
EXTRA_CLAUSE
FLAGS

APP_TBL must be set to the fully qualified Informix name of the desired apparition table ("my_input_db@my_input_server:my_apparition_table") and TEST_DB must be set to the fully qualified Informix name of the database to create output tables in ("my_output_db@my_output_server"). Note that srcgen does not create final output tables; however, it does create test tables with schemas identical to actual output tables in order to verify user input. This means that the user must have been granted RESOURCE permission for the specified Informix database. All test tables are dropped by srcgen so as not to pollute the Informix database with empty tables.

Next, the NAME_KEY environment variable should be set to a short string which is descriptive of the particular WAX executable being built. For the purposes of this example, set NAME_KEY to "simple".

EXTRA_CLAUSE specifies SQL clauses to append to the end of each generated CREATE TABLE statement, and allows output tables to be partitioned across multiple disks. The database administrator should determine what to set this environment variable to.

Finally, FLAGS is used to pass options to srcgen. Note that any of the standard retrieval or output columns can be renamed - see this section for details on all supported srcgen options.

The edited version of gen.sh file should look like this :


#!/bin/sh

APP_TBL="my_input_db@my_input_server:my_apparition_table"
TEST_DB="my_output_db@my_output_server"
NAME_KEY="simple"
EXTRA_CLAUSE=""
FLAGS="-v -name ${NAME_KEY}"

../bin/srcgen $APP_TBL $TEST_DB retrieve.tbl output.tbl grpapp.tbl . $FLAGS

Generating source code and compiling WAX

The source code generator can now be invoked by running the gen.sh script :


% gen.sh

This step generates all the source code and ancillary files needed to produce a WAX executable, including a GNU compatible make file, invoked via make_simple.sh. These various files are discussed further in the srcgen documentation.

WAX can now be compiled as follows :


% make_simple.sh

This produces a WAX executable named wax_simple. The arguments, behaviour, and options of WAX executables are detailed below, and a more complicated and realistic example is presented in the WAX plug-in guide.

WAX executable

Usage

wax -help

Returns detailed usage information.

wax -drop <database> <info table> <link table> [-single <single table>] [-grpapp <grouped apparition table>]

Drops the given tables from the specified database.

wax -create <database> <info table> <link table> [-single <single table>] [-grpapp <grouped apparition table>]

Creates the given output tables in the specified database.

wax -pri <band range> <band file> <group radius> <stitch dir> <app table> <database> <info table> <link table> [options]

Finds and stores groups with centroids inside the primary bands (bands with even indexes) in the given range.

wax -sec <band range> <band file> <group radius> <stitch dir> <app table> <database> <info table> <link table> [options]

Finds and stores groups with centroids inside the secondary bands (bands with odd indexes) in the given index range.

wax -band <dec min> <dec max> <group radius> <app table> <database> <info table> <link table> [options]

Finds and stores groups with centroids inside the given declination range. Note that in this mode, output tables which do not exist are automatically created.

Command Line Arguments

<database>

The fully qualified name of the Informix database to use for dropping, creating, and updating output tables (database@server).

Example: "ipac@rmt_gravel"

Since the '@' character often has special significance for UNIX shells, the argument should be enclosed in quotes.

<info table>

The name of the Informix output table to store group information (attributes) in.

<link table>

The name of the Informix output table to store group-apparition links in.

<band range>

The index(es) of the band(s) to process with the swiss cheese algorithm.

Examples : 30
           30-33

<band file>

The name of the file containing the sky subdivision to use (i.e. the mapping between declinations and band indexes).

<group radius>

The group radius to use (in arcseconds). This must be a floating point number between 0.1 and 5.0 (inclusive).

<stitch dir>

The name of the directory in which primary bands will store and secondary bands will read top and bottom group lists and apparition sets.

<app table>

The fully qualified name of the Informix table to retrieve apparitions from (database@server:table).

Example: "tmass@rmt_pebble:pt_src_01"

Since the '@' and ':' characters often have special significance for UNIX shells, the argument should be enclosed in quotes.

<dec min>

The minimum centroid declination of the groups to find and store. The value must be a floating point number in the range [-90.0, 90.0].

<dec max>

The maximum centroid declination of the groups to find and store. value must be a floating point number in the range [-90.0, 90.0].

Optional Arguments

[-v]

When specified, this flag causes wax to print out verbose comments when running.

[-p]

When specified, this flag causes wax to profile the execution time of various steps in the grouping process.

[-o]

When specified, this flag causes wax to use smaller spatial bins (internally), making the grouping process more effecient for extremely dense apparition tables.

[-d <density radius>]

When specified, this flag causes wax to use the given radius (in arcseconds) as the density radius. The density radius must be less than or equal to the group radius.

[-filter <SQL filter>]

When specified, wax will only retrieve apparitions satisfying the given SQL filter, which must be a valid SQL boolean expression involving columns contained in the apparition table.

[-single <single table>]

The name of the Informix output table to store single apparition groups (singletons) in. When this option is specified, singletons will be omitted from the group information table and group-apparition link table.

[-grpapp <grouped apparition table>]

The name of the Informix output table to store grouped apparitions in. If unspecified, grouped apparitions are neither processed nor stored.

[-pad <pad multiple>]

The integer multiple of the group radius by which the apparition retrieval range for primary bands will be padded. The value must be an integer greater than or equal to 6, and is set to 10 if unspecified.

[-bsp <bsp directory>]

The name of the directory containing the scan coverage BSP files to read when performing 2MASS scan coverage computations. These are generated from ASCII scan corner files with the bspgen (BSP File Generator) helper application.

ASCII Table Files

This section of the documentation describes the format of all the ASCII table files used by the WAX executable and the helper applications.

An ASCII table file consists of a series of lines (terminated by the ASCII new line character, 0x0A). Lines are interpreted as follows

empty lines are skipped
lines consisting entirely of whitespace are skipped
lines in which the first non-whitespace character is a '#' are interpreted as comment lines and are skipped.

Otherwise a line is assumed to correspond to a semi-colon delimited list of fields in which leading and trailing whitespace for each field is ignored :

field 1; field 2 [;field 3 ...]

Field values may contain embedded whitespace. The last field in a line should not be followed by a semi-colon, but can be followed by a '#' character, causing all remaining text in the line to be interpreted as a comment.

The following is an example ASCII table file demonstrating each of the features described. For the purposes of this example, whitespace is mapped to the _ character, and new lines are mapped to the ¶ symbol. The field delimiting semi-colon is displayed in red, comments are displayed in green, ignored whitespace in pink, and actual field value characters in blue:

#_An_example_retrieval_column_table_file__¶
__#_a_comment_line¶
#another_comment__line_____¶
¶
___a_field_value__;field_2__¶
_row2_field_1;;row2_field_3#_end_of_line_comment¶

or, without any special color coding :

# An example retrieval column table file
  # a comment line
#another comment  line

   a field value  ;field 2
 row2 field 1;;row2 field 3# end of line comment

Retrieval column file

The retrieval column table is an ASCII table file with a single column containing the names of the apparition table columns to retrieve. Each non-empty/non comment line (row) shall contain a single Informix column name. The standard retrieval columns ( ra, dec, and scan_key ) do not need to be listed in the table file.

The following is an example retrieval column file :

# An example retrieval column table file.
j_m  # 2MASS J magnitude
h_m  # 2MASS H magnitude
k_m  # 2MASS Ks magnitude

When the srcgen (Source Code Generator) helper application is run, the info.h header file is generated. This file will define a C structure, AppInfo, containing variables corresponding to the apparition database table retrieval columns. The variable names will be identical to the retrieval column names. Furthermore, if a database table column is marked as allowing NULL values, the AppInfo structure will also contain an indicator variable named by appending "_ind" to the retrieval column name.

Consider the example table above - assuming that the database table containing the j_m, h_m, and k_m columns allows these columns to contain NULL values and declares them to be of SQL type float, the generated C structure would look like this :

struct app_info_t
{
    double j_m;
    double h_m;
    double k_m;
    short  j_m_ind;
    short  h_m_ind;
    short  k_m_ind;
};

For details on how Informix SQL column types are mapped to C variable types, see the Type mapping between Informix and C section.

If no retrieval columns are specified, a structure containing a single dummy variable is generated :

struct app_info_t
{
    char dummy;
};

An indicator variable $v_{ind}$ for a C variable $v$ is always of type short. When $v_{ind} = -1$ , the value of $v$ must be interpreted as NULL. If $v_{ind} = 0$ , then the value of $v$ is valid.

Note that the C variables in AppInfo (except indicators) appear in order of their type sizes. This stops the compiler from padding the structure to satisfy data alignment requirements. Indicator variables always appear last in the structure.

Output column file

The output column table is an ASCII table file. Each non-empty/non comment line (row) shall contain two or three fields. The first shall be the Informix name of a column in the group information table, the second the Informix SQL type specification of the named column, and the optional third field shall be a single character flag indicating whether or not the output column should be included in the singleton table. A flag character equal to one of [yYtT1] will include the column in the singleton table, whereas a character in [nNfF0] will exclude it. If this flag is unspecified, the column is included by default.

The allowed column type specifications are described here, and are illustrated below:

# An example output column table file
  ra             ; decimal(15;6) NOT NULL ; y
  dec            ; decimal(15;6) NOT NULL ; y
  sep_avg        ; decimal(15;6) NOT NULL ; t
  sep_sig       ;  decimal(11;9) NOT NULL ; t
  use_cntr      ;  integer                ; y
  smin        ;    smallint NOT NULL ; 1
  smax        ;    smallint NOT NULL ; y
  spos        ;    smallint NOT NULL ; Y
  j_m_avg     ;  decimal(7;4) ; n
  h_m_avg     ;  decimal(7;4) ; N
  k_m_avg     ;  decimal(7;4) ; f
  j_mchisq   ; decimal(15;6) ; F
  h_mchisq   ; decimal(15;6) ; 0
  k_mchisq   ; decimal(15;6) ; n
  x           ; float NOT NULL; y
  y           ; float NOT NULL; Y
  z           ; float NOT NULL; T
  spt_ind        ; integer  NOT NULL; t

When the srcgen (Source Code Generator) helper application is run, the info.h header file is generated. This file will define a C structure, GroupInfo, containing variables corresponding to the output database table columns. The variable names will be identical to the output column names. Furthermore, if an output column type specification does not include NOT NULL, the GroupInfo structure will also contain an indicator variable named by appending "_ind" to the output column name.

For the example table above the generated C structure would look like this :

struct group_info_t
{
    double ra;
    double dec;
    double sep_avg;
    double sep_sig;
    double j_m_avg;
    double h_m_avg;
    double k_m_avg;
    double j_mchisq;
    double h_mchisq;
    double k_mchisq;
    double x;
    double y;
    double z;
    int    use_cntr;
    int    spt_ind;
    short  smin;
    short  smax;
    short  spos;

    short  j_m_avg_ind;
    short  h_m_avg_ind;
    short  k_m_avg_ind;
    short  j_mchisq_ind;
    short  h_mchisq_ind;
    short  k_mchisq_ind;
    short  use_cntr_ind;
};

For details on how Informix SQL column types are mapped to C variable types, see the Type mapping between Informix and C section.

If no output columns are specified, a structure containing a single dummy variable is generated :

struct group_info_t
{
    char dummy;
};

An indicator variable $v_{ind}$ for a C variable $v$ is always of type short. When $v_{ind} = -1$ , the value of $v$ is interpreted as NULL. Otherwise, if $v_{ind} = 0$ , the value of $v$ is assumed to be valid.

Note that the C variables in GroupInfo (except indicators) appear in order of their type sizes. This stops the compiler from padding the structure to satisfy data alignment requirements. Indicator variables always appear last in the structure.

Grouped apparition column file

The grouped apparition column table is an ASCII table file. Each non-empty/non comment line (row) shall contain two fields. The first shall be the Informix name of a column in the grouped apparition table, and the second the Informix SQL type specification of the named column.

The allowed column type specifications are described here, and are illustrated below:

# An example grouped apparition column table file
  gcntr      ;  integer NOT NULL
  smin        ;    smallint NOT NULL
  smax        ;    smallint NOT NULL
  spos        ;    smallint NOT NULL

When the srcgen (Source Code Generator) helper application is run, the info.h header file is generated. This file will define a C structure, GroupedAppInfo, containing variables corresponding to the grouped apparition database table columns. The variable names will be identical to the grouped apparition column names. Furthermore, if a grouped apparition column type specification does not include NOT NULL, the GroupedAppInfo structure will also contain an indicator variable named by appending "_ind" to the grouped apparition column name.

For the example table above the generated C structure would look like this :

struct grouped_app_info_t
{
    long   gcntr;
    short  smin;
    short  smax;
    short  spos;
};

For details on how Informix SQL column types are mapped to C variable types, see the Type mapping between Informix and C section.

If no output columns are specified, a structure containing a single dummy variable is generated :

struct grouped_app_info_t
{
    char dummy;
};

Note that the C variables in GroupedAppInfo (except indicators) appear in order of their type sizes. This stops the compiler from padding the structure to satisfy data alignment requirements. Indicator variables always appear last in the structure.

Band file

The band file an ASCII table file with a single column containing strictly increasing declination values (assumed to be floating point numbers and expressed in degrees). Each non-empty/non comment line (row) shall contain a single declination. Furthermore, each successive declination value shall be strictly greater than the last value. These ascending declinations, $d_0,d_1,d_2$

... are mapped to bands as follows :

The first band , spans
The second band , spans
...

Note that the swiss cheese algorithm places a restriction on how skinny a declination band can be - this is checked at run-time, and depends on the group radius as well as the padding multiple used by the WAX executable reading the band file.

Type mapping between Informix and C

Informix SQL type	C language type	Comments
`CHAR(n) [NOT NULL]`	`char[]`	The C `char` array is a null-terminated version of the database field and is therefore an array of length `n+1`.
`DATE [NOT NULL]`	`long`	The encoding of a date in a `long` is defined by Informix. See the Informix ESQL/C API documentation for functions which manipulate dates.
`SMALLINT [NOT NULL]`	`short`
`INT [NOT NULL], SERIAL [NOT NULL]`	`int`
`SMALLFLOAT [NOT NULL]`	`float`
`FLOAT [NOT NULL]`	`double`
`DECIMAL(M,N) [NOT NULL]`	if `M<7: float`, `double` otherwise
`SERIAL8 [NOT NULL]`	`-`	currently unsupported
`INT8 [NOT NULL]`	`-`	currently unsupported
`DATETIME [NOT NULL]`	`-`	currently unsupported
`INTERVAL [NOT NULL]`	`-`	currently unsupported
`MONEY [NOT NULL]`	`-`	currently unsupported

Generated on Thu Oct 21 13:19:38 2004 for WAX Version 2.1 by