Appendix G. Using GoldenGate Data Conversion Services

This appendix describes how to use the GoldenGate data conversion services. It covers these topics:

Converting Data Using the GoldenGate Data Conversion Service

This section describes how you can use the GoldenGate data conversion service in your application. Specifically, it explains:

Overview of the Conversion Process

To convert data using GoldenGate, follow these steps:

  1. Choose a converter.

    • Obtain a list of converters that read the source format and write the target format.

    • Create a conversion context structure and set conversion parameters.

    • Evaluate the list of converters to determine which one is best suited for the current conversion.

  2. Convert data.

    There are two methods of converting data, depending on whether the data is in a stream or in a file.

    Converting Stream Data

    • Initialize the selected converter.

    • Send data through the converter and read results back.

    • Clean up resources by destroying the conversion context.

    Converting Data Files

    • Call the file conversion function.

    • Clean up resources by destroying the conversion context.

The Converter Registry

GoldenGate maintains a list of available converters in the converter registry. This registry contains an entry for each converter, specifying characteristics such as the type of input data it takes and the type of output data it produces. To find out if there are any converters that will convert from format “A” to format “B,” you can query the registry.

GoldenGate returns a list of converters that take the specified input and produce the specified output. You can be as specific as you like when querying the registry, to ensure that only relevant converters are listed. You should also use the query to eliminate inappropriate categories of converter, such as those of type StreamToStream if you are converting a file. If the list contains more than one converter, you may need to evaluate the converters to see which one best meets your needs. Even if the list contains only one converter, you should evaluate it to make sure it can handle your conversion request.

Creating a Conversion Context

To communicate with a converter, you must create a conversion context. The conversion context is a data object that stores conversion parameters. The conversion context is passed to subsequent library calls that set input and output parameters, evaluate converters, initialize the conversion pipeline, and move data through it.

Once you have created a conversion context and specified the desired conversion parameters, you can evaluate the list of converters you obtained when you queried the registry. For example, suppose you want to convert from one audio format to another and change the sample rate at the same time. Querying the registry returns a list of converters that will convert between the specified input and output formats. To determine if any of these converters will perform the desired sample rate conversion, you have to create a conversion context, set the desired parameters (including input and output sample rate) and then evaluate the individual converters.

Evaluating a Converter

It's best to evaluate a converter before you invoke it to perform a conversion. You do this for the following reasons:

  • Evaluation gives the converter an opportunity to inspect your data parameters. Some converters will have more functionality than others, even though their input and output types are the same. A well-designed converter will know just by looking at parameters whether it can do the conversion.

  • Conversion is typically an expensive operation. If your attempt to convert fails, you can still choose a different converter and try again, but you could have avoided lost time by trying a converter that can accept your specific request.

Depending on your needs, you can select the first converter on the list that passes the evaluation stage, or evaluate the whole list and use your own rules to choose between those that pass.

Converting Data In a File or Stream

Once you determine the converter to use, the final stage depends on whether you are converting data in a file or a stream.

If you are converting a stream, initialize a conversion pipeline that reads your stream and passes back results as they are available. Then you send all your data through the pipeline and read the results until you see the end of stream marker for the pipeline. At this point, terminate the pipeline. This causes GoldenGate to clean up data structures it keeps for maintaining a stream conversion.

If you are converting a file, the procedure is simpler. You call a single GoldenGate function to perform the operation, and wait for results. If necessary you can provide a callback function that will notify you when results become available. This allows you to service other events going on in your application during what may be a long conversion.

Selecting a Converter

This section describes how to select a converter by querying the converter registry and setting up the conversion context. Specifically, this section covers:

Querying the Converter Registry

To query the converter registry, you specify a set of constraints. Each constraint consists of an attribute (such as input format), a value for the attribute, and a comparison operator. For example, you can ask for a converter that has input format equal to “AIFF_FILE,” and version number greater than 2. Use the SgCvtSetQueryConstraint() function to fill in an array of SgCvtQueryConstraint structures, then pass the array to the SgCvtQueryRegistry() function. The following code fragment demonstrates a simple query that locates converters capable of converting AIFF_FILE to WAVE_FILE:

SgCvtQueryConstraint constraints[2];
SgCvtStatus status;
SgCvtConverterId *converters;
int num_constraints, num_converters;
SgCvtRegistry registry = NULL;
status = SgCvtSetQueryConstraint(constraints[0],
         SG_CVT_ATTR_INPUT_TYPE, "AIFF_FILE", SG_CVT_OP_EQ);
status = SgCvtSetQueryConstraint(constraints[1],
         SG_CVT_ATTR_OUTPUT_TYPE, "WAVE_FILE", SG_CVT_OP_EQ);
num_constraints = 2;
status = SgCvtQueryRegistry(constraints, num_constraints,
            &registry, &converters, &num_converters);

The SgCvtQueryRegistry() function returns an array of converter IDs that can be used to identify the individual converters.

The registry argument specifies the GoldenGate converter registry to be queried. During this call, the registry is located on disk (/var/GoldenGate/ConverterRegistry by default), and its contents parsed to find a converter that matches your requirements.

The first time you call SgCvtQueryRegistry, specify registry as NULL as in the previous example, which causes this lookup. When you have finished converting, you can either call SgCvtFreeRegistry to release the resources that GoldenGate may have cached after reading the file, or you can re-use the value returned in registry for subsequent queries, avoiding the overhead of looking up the file.

If you choose to free the registry between queries, your program will always have the latest information, even if the registry changes while your program is running. If you choose to re-use the registry handle, you have no control over whether or not GoldenGate will re-parse the registry. It will try to use its cache first. If for any reason the cache is invalid, GoldenGate may at its discretion rebuild it by reading the disk-based registry again.

The converters argument returns an array of matching converter IDs, of which the first num_converters are valid and matched the query. You should free this array when you are finished using it, using free(3).

Table G-1 lists the attributes you can query.

Table G-1. Converter Attributes

Attribute Name

Description

SG_CVT_ATTR_NAME

Converter name

SG_CVT_ATTR_INPUT_FORMAT

Input format

SG_CVT_ATTR_OUTPUT_FORMAT

Output format

SG_CVT_ATTR_IO_METHOD

StreamToStream or FileToFile

SG_CVT_ATTR_INPUT_LABEL

Input format, human readable version

SG_CVT_ATTR_OUTPUT_LABEL

Output format, human readable version

SG_CVT_ATTR_VENDOR

Vendor's name

SG_CVT_ATTR_VERSION

Vendor's version information

SG_CVT_ATTR_DESCRIPTION

Description of converter

Most of the time, you'll be interested in the input format and output format attributes. “Supported Target Formats” in Chapter 7 lists common data formats. Other attributes may be useful when listing converters for users. For example, if you want the user to choose between two converters that perform the same conversion, you can display the vendor names and version numbers.

Table G-2 lists the operators you can use in your query.

Table G-2. Query Operators

Operator

Symbol

equal to

SG_CVT_OP_EQ

not equal to

SG_CVT_OP_NE

less than

SG_CVT_OP_LT

less than or equal to

SG_CVT_OP_LE

greater than

SG_CVT_OP_GT

greater than or equal to

SG_CVT_OP_GE

Note that if more than one constraint is specified on a single attribute, a logical AND is implied. For example, you can select a range of version numbers by setting “version greater than or equal to one” as one constraint and “version less than or equal to three” as a second constraint.

SgCvtQueryConstraint can return the following status value:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SgCvtQueryRegistry can return the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_FAILURE
 

Could not find the registry, or failed to parse it. Most likely when the default registry has been edited to add new converters, and a syntax error introduced. You may also be loading the wrong file. Make sure that if there is a file called ConverterRegistry on your path, it is a valid registry using the CDF syntax. Also make sure the CVT_REGISTRY_OVERRIDE variable is not set.

Setting Up the Conversion Context

Before you can evaluate or use a converter, you must create a conversion context and set parameters governing the conversion. Use the SgCvtCreateConversionContext() function to create a conversion context:

SgCvtStatus
SgCvtCreateConversionContext(SgCvtConversionContext *context)

SgCvtCreateConversionContext can return the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_NOMEM
 

Insufficient memory to allocate a context.

Next, set any digital media parameters that affect your conversion by calling SgCvtSetContextInfo.

SgCvtStatus SgCvtSetContextInfo
    (
    SgCvtConversionContext    context,
    unsigned long             valuemask,
    SgCvtContextInfo          *context_data
    );

where

context  

specifies the context you created with SgCvtCreateConversionContext 

valuemask  

specifies which fields in the SgCvtContextInfo structure are being set in the context. This is specified as any of the following OR'ed together:

SG_CVT_INFO_INPUT_PARAMS
SG_CVT_INFO_OUTPUT_PARAMS
SG_CVT_INFO_META_PARAMS
SG_CVT_INFO_INPUT_FILE
SG_CVT_INFO_INPUT_HOST
SG_CVT_INFO_OUTPUT_FILE
SG_CVT_INFO_OUTPUT_HOST

 context_data  

specifies the values being set

SgCvtSetContextInfo can return the following status value:

SG_CVT_E_SUCCESS
 

The operation succeeded.

See the IRIS Media Libraries Programming Guide for information on setting DMparams.

Evaluating Converters

To evaluate a converter, call SgCvtEvaluateConverter():

SgCvtStatus 
SgCvtEvaluateConverter(SgCvtConverterId converter_id, 
                       SgCvtConversionContext context
                       DMparams **output_params)

where

converter_id 

is a converter ID returned by the SgCvtQueryRegistry() function

context 

is a valid conversion context obtained from SgCvtCreateConversionContext() 

output_params 

returns the output of the request. Converters may set these parameters, even though they accept the request.

SgCvtEvaluateConverter() can return the following status values:

SG_CVT_E_ACCEPT
 

The converter can perform the conversion specified by the conversion context.

SG_CVT_E_REJECT
 

The converter can't perform the requested conversion.

When evaluating a converter returns a status of SG_CVT_E_ACCEPT, you should take one final step before calling the converter. You should inspect the output_params argument, which returns a DMparams list describing the result that the converter will produce. If your program has very strict requirements, this will help protect you if the converter has accepted the request but cannot honor what it considers a minor parameter, or if you passed a parameter it could not understand.

Getting Converter Details

If your program needs to display information about available converters, or do other processing based on the data stored about a converter in the converter registry, call SgCvtGetConverterAttributes() to get a description of it. The function prototype for SgCvtGetConverterAttributes() is shown below.

SgCvtStatus 
SgCvtGetConverterAttributes(SgCvtConverterId converter_id,
                           unsigned long converter_attr_mask,
                           SgCvtConverterAttrs *attributes)

When you are finished using the fields of the SgCvtConverterAttrs structure, you should free the string attributes and the structure itself (if you allocated it dynamically) using free(3C).

SgCvtGetConverterAttributes can return the following status value:

SG_CVT_E_SUCCESS
 

The operation succeeded.

Using GoldenGate to Convert Data

This section describes the different methods you can use to convert data. Topics include:

Converting Data Using File Converters

Your file-based data is always converted using the function SgCvtConvertFileToFile. Before you call it however, you need to decide whether you want the function to block while the conversion is going on, or return immediately and let you know later that the conversion is complete.

In many cases blocking mode is sufficient, and it is much simpler to use if your program is not naturally event driven. However, if your application has a GUI, you may prefer non-blocking mode because it allows your event loop to keep running while conversion is going on. When conversion is complete, you are notified through a callback function that you supply, and you can use the converted data.

Both modes are invoked using SgCvtConvertFileToFile:

typedef void (*SgCvtCallback)(SgCvtConversionContext context,
                              void *client_data,
                              void *callback_data);
     
SgCvtStatus SgCvtConvertFileToFile
        (
         SgCvtConversionContext   context,
         SgCvtConverterId         converter_id,
         char                     *input_file,
         char                     *output_file,
         unsigned long            callback_mask,
         SgCvtCallback            callback,
         void                     *client_data
         );

where

context  

the conversion context, holding the I/O filenames and parameters

converter_id  

the converter ID, returned by SgCvtQueryConverter

input_file 

pathname of input file. You must have read permission.

output_file  

pathname of output file. You must have write permission.

callback_mask 

mask indicating when callback should be called. It should be some logical combination of the following values:

SG_CVT_CB_FLAG_CONVERSION_DONE
      (after completion)
SG_CVT_CB_FLAG_STAGE_DONE
      (after each stage if multi-stage pipeline)

callback 

specifies the callback function

client_data 

a pointer to application-defined data structure that will be passed to the callback when invoked

If specified, the callback argument is the address of the function to call when conditions specified by the callback_mask arise. If the callback function is not specified, or the mask is zero, the function executes in blocking mode.

SgCvtConvertFileToFile can return the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_BAD_CONVERTER_TYPE
 

The converter was not registered as FileToFile IO method.

SG_CVT_E_READ_FAILED
 

The input file could not be read. It may be missing, or the permissions are insufficient for reading.

SG_CVT_E_WRITE_FAILED
 

The output file could not be written. This can happen if the user does not have write permission for the target directory, or if the supplied pathname was invalid.

Converting Data Using Stream Data Converters

To convert data using your specified converter, you must initialize the conversion pipeline, and then send the data through. After reading the last block of converted data, clean up by destroying the conversion context to free the resources associated with the pipeline. This section covers the following topics:

Initializing the Pipeline

Prepare the converter to receive data by calling SgCvtInitializePipeline():

SgCvtStatus 
SgCvtInitializePipeline(SgCvtConversionContext context,
                        SgCvtConverterId  converter_id)

where

context 

is a valid conversion context obtained from SgCvtCreateConversionContext()

converter_id 

is a converter ID returned by the SgCvtQueryRegistry() function

SgCvtInitializePipeline can return the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_FAILURE
 

The context or its contents is bad or one of the subprocesses required to host a converter function could not be launched.

SG_CVT_E_BAD_CONVERTER_TYPE
 

The converter was not registered as StreamToStream. Converters that are designed to work with streaming data advertise themselves as using the StreamToStream method of I/O in the registry.

Sending and Receiving Data

You may send and receive arbitrarily sized blocks of data, so use a block size that is convenient.

Send data to the converter using SgCvtSendData(). The function prototype for SgCvtSendData() is shown below:

SgCvtStatus  SgCvtSendData(
    SgCvtConversionContext context,
    void                   *data,
    size_t                 length,
    DMparams               *params,
    boolean_t              canwait
)

where

context 

is a valid conversion context

data 

is a pointer to the data block to be converted

length 

is the length of the data block

params 

is a DMparams structure describing the data to be converted

canwait 

is a boolean value that indicates what the function should do if it cannot send the data immediately. If you specify B_TRUE, SgCvtSendData() will block until it can send the data to the conversion pipeline. If you specify B_FALSE, SgCvtGetData() will return immediately with a status of SG_CVT_E_AGAIN. This status indicates that you should try again.

SgCvtSendData can return with the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_FAILURE
 

An I/O error occurred while trying to send data through the pipe connecting two pipeline components.

SG_CVT_E_AGAIN
 

Required resources were temporarily unavailable. The caller should retry later.

Read data from the converter using SgCvtGetData(). The function prototype for SgCvtGetData() is shown below:

SgCvtStatus  SgCvtGetData(
    SgCvtConversionContext context,
    size_t                 buf_len,
    void                   *buffer,
    size_t                 *length_returned,
    DMparams               **params_returned,
    boolean_t              canwait
)

where

context 

is a valid conversion context

buf_len 

specifies the size of buffer

buffer 

is a pointer to a pre-allocated buffer of at least buf_len bytes.

length_returned 

is the actual length of the returned data (this may be less than bytes_requested if non-blocking mode is specified, or if the converter encounters end-of-stream) SgCvtGetData

params_returned 

is a DMparams structure describing the converted data.

canwait 

is a boolean value that indicates what the function should do if no data is available. If you specify B_TRUE, SgCvtGetData() will block until data becomes available from the conversion pipeline. If you specify B_FALSE, SgCvtGetData() will return immediately with a status of SG_CVT_E_QUEUE_EMPTY. This status indicates that you should try again.

SgCvtGetData can return the following status values:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SG_CVT_E_FAILURE
 

An I/O error occurred while trying to read data from the pipe connecting two pipeline components.

SG_CVT_E_AGAIN
 

Required resources were temporarily unavailable. The caller should retry later.

SG_CVT_E_END_OF_STREAM
 

The operation succeeded, and the end of the data has been reached.

The non-blocking mode of SgCvtSendData() and SgCvtGetData() allows programs to continue working on other tasks (such as handling events from a graphical interface) while waiting to send data to or read data from the conversion pipeline.

Cleaning Up

When you've sent the last of the data to the converter, call SgCvtSendEndOfStream() to indicate the end of the data. After you've read the last of the converted data, free the resources associated with the conversion context by calling SgCvtDestroyConversionContext():

SgCvtStatus SgCvtSendEndOfStream(
    SgCvtConversionContext context
);

SgCvtStatus  SgCvtDestroyConversionContext(
    SgCvtConversionContext context
)

If you need to terminate the conversion process before reaching the end of the data, call SgCvtDestroyConversionContext().

SgCvtSendEndOfStream can return the following status value:

SG_CVT_E_SUCCESS
 

The operation succeeded.

SgCvtDestroyConversionContext can return the following status value:

SG_CVT_E_SUCCESS
 

The operation succeeded.

Converter Return Status Values

Table G-3 lists converter functions and their return status values.

Table G-3. Converter Return Status Values

Function

Return Value

Description

SgCvtSetQueryConstraint

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtQueryRegistry

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_FAILURE

Could not find the registry, or failed to parse it. Most likely when the default registry has been edited to add new converters, and a syntax error introduced. You may also be loading the wrong file. Make sure that if there is a file called ConverterRegistry on your path, it is a valid registry using the converter description file syntax. Also make sure the CVT_REGISTRY_OVERRIDE variable is not set.

SgCvtGetConverterAttributes

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtCreateConversionContext

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_NOMEM

Insufficient memory to allocate a context.

SgCvtDestroyConversionContext

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtSetContextInfo

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtGetContextInfo

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtEvaluateConverter

SG_CVT_E_ACCEPT

The converter can perform the requested conversion.

 

SG_CVT_E_REJECT

The converter cannot perform the requested conversion.

SgCvtInitializePipeline

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_FAILURE

The context or its contents is bad or one of the subprocesses required to host a converter function could not be launched.

 

SG_CVT_E_BAD_
CONVERTER_TYPE

The converter was not registered as StreamToStream. Converters that are designed to work with streaming data advertise themselves as using the StreamToStream method of I/O in the registry.

SgCvtTerminatePipeline

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtSendData

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_FAILURE

An I/O error occurred while trying to send data through the pipe connecting two pipeline components.

 

SG_CVT_E_AGAIN

The required resources were temporarily unavailable. The caller should retry later.

SgCvtGetData

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_FAILURE

An I/O error occurred while trying to read data from the pipe connecting two pipeline components.

 

SG_CVT_E_AGAIN

The required resources were temporarily unavailable. The caller should retry later.

 

SG_CVT_E_END_OF_
STREAM

The operation succeeded, and the end of the data has been reached.

SgCvtSendEndOfStream

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtEncodeParams

SG_CVT_E_SUCCESS

The operation succeeded.

SgCvtDecodeParams

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_NOMEM

Insufficient memory to allocate structures.

SgCvtFreeEncodedParams

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_FAILURE

The data could not be decoded.

SgCvtConvertFileToFile

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_BAD_
CONVERTER_TYPE

The converter was not registered as FileToFile I/O method.

 

SG_CVT_E_READ_
FAILED

The input file could not be read. It may be missing, or the permissions are insufficient for reading.

 

SG_CVT_E_WRITE_
FAILED

The output file could not be written. This can happen if the user does not have write permission for the target directory, or if the supplied pathname was invalid.

SgCvtGetFileSelectionTarget

SG_CVT_E_SUCCESS

The operation succeeded.

 

SG_CVT_E_UNKNOWN _
TYPE

The file type could not be determined.

 

SG_CVT_E_NO_TARGE T

The selection target for the type of file could not be determined, or there is none.

 

SG_CVT_E_FAILURE

The operation could not be performed for another reason, such as the underlying file typing database library could not be accessed, or the database itself was corrupt or missing.

SgCvtIsPipeline

B_TRUE

The translator is a multi-stage pipeline.

 

B_FALSE

The translator is a single-stage converter.


Compiling and Linking Your Program with GoldenGate

To compile and link your program, you need to include the header file SgCvt.h and include the library libcvt in your link line.

An example of a simple GoldenGate program follows. It includes the required header file, enumerates the registered converters, and prints their input and output labels.

#include <SgCvt.h>
main(int argc, char **argv)
{
    SgCvtRegistry         registry = NULL;
    int                   n=0;
    SgCvtStatus           s;
    SgCvtConverterId      *cvtrs;
    int                   ncvtrs;

    s = SgCvtQueryRegistry(NULL, 0, &registry, &cvtrs, &ncvtrs);
    for (n=0; n<ncvtrs; n++) {
            SgCvtConverterAttrs attrs;

            SgCvtGetConverterAttributes(cvtrs[n],
                 SG_CVT_ATTR_FLAG_INPUT_LABEL |
                 SG_CVT_ATTR_FLAG_OUTPUT_LABEL,
                 &attrs);

            printf(“%d %25s -> %s\n”,
                 n+1, attrs.input_label, attrs.output_label);

            free(attrs.input_label);
            free(attrs.output_label);
     }

     SgCvtFreeRegistry(registry);
}

The following Makefile illustrates the compile and link requirements for this program.

#
# Makefile for GoldenGate Listing sample program
#
CC      =       cc
TARGET  =       gg_listing
SOURCES =       $(TARGET).

INCLUDES=       -I/usr/include/convert

REQLIBS =       -lcvt
all:
        $(CC) -o $(TARGET) $(INCLUDES) $(SOURCES) $(REQLIBS)

Writing Converters for the GoldenGate Data Conversion Service

This section describes how to write converters that can integrate with GoldenGate and become available to any component that is aware of GoldenGate. The following information assumes that you are familiar with the interfaces described in “Converting Data Using the GoldenGate Data Conversion Service”. Both converters and applications use many of the functions and data structures.

Overview of the Converter Writing Process

Creating a GoldenGate data converter involves writing the converter and building the DSO, then testing, registering, and installing the converter. The topics below describe:

“Some Sample Converters” shows annotated code for two converters.

Writing Converter Code

This section describes how to write converter code and includes the following topics:

Implementing Your Converter - Handling Evaluation Requests

When the operation field of the SgCvtConverterData structure passed to your converter is equal to SG_CVT_REQ_EVALUATE, your converter should inspect the input, output, and meta parameters held in the conversion context and determine whether or not it can satisfy the request, without actually performing conversion.

If your converter can satisfy the request, it should set the status_return field of the SgCvtConverterData structure to SG_CVT_E_ACCEPT before returning. Otherwise it should set status_return to SG_CVT_E_REJECT.

Implementing Your Converter - Handling Conversion Requests

When the operation field of the SgCvtConverterData structure is equal to SG_CVT_REQ_CONVERT, your converter must extract the necessary information from the SgCvtConverterData structure it is passed, and perform the conversion if possible. If conversion is successful, it should return with the status_return field set to SG_CVT_E_SUCCESS, and if it is unsuccessful, the status_return field should be set to either SG_CVT_E_FAILURE or a more specific error code if appropriate (see the error codes available in SgCvt.h).

How the converter is implemented depends on whether you are writing the conversion code yourself, or simply using an existing command-line converter.

If you are creating a “wrapper” to make an existing UNIX command available through the GoldenGate conversion service, the procedure is quite straightforward.

In this case, your converter is a function that gathers the input and output requirements from its arguments, and executes the external UNIX command (for instance, by calling the system(2) function).

Your function should do as much checking as possible to ensure that the external command can work. For instance, you should verify that the command is installed before calling it, and that you have execute permission.

Also check for appropriate permissions to read input files and write output files, in the case of file converters. If you detect an error before calling the command, return an error status in the status field of the data argument.

For example, the code below shows a FileToFile converter that wraps an existing UNIX command rtf2html. You will find other fully annotated examples at the end of this section.

#include <libgen.h>
#include <SgCvt.h>

void RtfToHtml(void *arg)
{
   SgCvtConverterData *data = (SgCvtConverterData *) arg;
   SgCvtContextInfo ctx_info;
   char cmd[BUFSIZ];
   int sys_status = 0;
   char *xlator_path;

   /* Evaluation - just accept for this example */
   if (data->operation == SG_CVT_REQ_EVALUATE) {
      data->status_return = SG_CVT_E_ACCEPT;
      return;
   }
   
   /* Conversion */

   /* depends on `rtf2html' command being available */
   xlator_path = pathfind(getenv (“PATH”), “rtf2html”, “rx”);
   if (xlator_path == NULL) {
      data->status_return = SG_CVT_E_MISSING_COMMAND;
      return;
   }

   (void) SgCvtGetContextInfo(data->context,
                SG_CVT_INFO_INPUT_FILE          |
                SG_CVT_INFO_OUTPUT_FILE,
                &ctx_info);

   /* cmd syntax is `rtf2html inputfile outputfile'
   sprintf(cmd, “%s %s %s 2> /dev/null”, xlator_path, 
           ctx_info.input_file, ctx_info.output_file);

   sys_status = system(cmd);

   data->status_return =
      sys_status ? SG_CVT_E_FAILURE : SG_CVT_E_SUCCESS;

   return;
}

Notice that GoldenGate passes the necessary information to a converter by reference. The SgCvtConverterData structure is the mechanism for this. It is defined as follows:

typedef struct {
        SgCvtRequestType        operation;
        SgCvtConversionContext  context;
        DMparams                *output_params;
        SgCvtStatus             status_return;
} SgCvtConverterData;

If your converter does not use an external command to translate the data, but does the conversion itself, the structure of the converter function is essentially the same.

You still use the SgCvtConverterData structure to communicate with GoldenGate. Between extracting the necessary arguments from the structure and returning from the function, you just call your own functions that do the conversion.

Input and Output Formats

Your converter should use standardized names for its input and output types wherever possible. This is important because applications are written to request data by a particular name. If your converter uses a different name for the same data format, GoldenGate will not find your converter and the conversion may fail.

See “Supported Target Formats” in Chapter 7 for the data formats supported by the default Silicon Graphics converters.

You can also use your own data format names. However, the name your application uses must match the name you registered so GoldenGate ca l find the converter. However, if you use your own data format names, it is unlikely that other applications will be able to take advantage of your converter. Do this only if the format name is well understood among all the applications you intend to cooperate with.

Process Blocking

You can use SgCvtGetData() and SgCvtSendData() in either blocking or non-blocking mode, depending on your requirements. Both modes are described in “Converting Data Using File Converters ”.

If your converter needs to return immediately to do other work, such as tracking activity on an I/O device, you should set the canwait argument to these functions to B_FALSE. If the conversion pipeline is not ready for an immediate read or write operation, the call will return immediately with a status value indicating that nothing happened and that you should try the same operation again. For additional information on the canwait argument, see “Sending and Receiving Data”.

If SgCvtSendData() cannot send data immediately and canwait is B_FALSE, it returns SG_CVT_E_AGAIN. This indicates that your data has not been sent, and you should try the operation again, using the same data.

SgCvtGetData() returns SG_CVT_E_QUEUE_EMPTY if there is no data immediately available and canwait is B_FALSE. You should try the operation again later.

If your converter has no other I/O requirements, you can simplify your code slightly by setting the canwait argument to B_TRUE. You should use this option by default, because it can eliminate redundant context switching to your idling converter, and improve system performance.

There are two categories of converter: FileToFile and StreamToStream.

A FileToFile converter uses the input and output file attributes of the conversion context to get its input and save its output, as shown in the example above.

A StreamToStream converter follows this general procedure after extracting the required parameters from the context:

  • Fetch a block of input data using SgCvtGetData

  • Convert the data to the new format

  • Send converted data back to GoldenGate

The converter repeats these steps until it receives a status of SG_CVT_E_END_OF_STREAM from SgCvtGetData, and it successfully sends all the converted data. Then it calls SgCvtSendEndOfStream to tell GoldenGate it is finished converting, and finally it returns.

The functions used for stream conversion are the same ones used by applications to work with conversion streams:

  • To fetch input and output parameters to be used in the conversion, use SgCvtGetContextInfo.

  • To fetch a block on data for conversion, use SgCvtGetData.

  • To send a block of converted data back to GOldenGate, use SgCvtSendData.

  • To break your connection to the stream and tell GoldenGate your converter is finished, use SgCvtSendEndOfStream.

Programming Constraints

Keep in mind the following constraints when writing converters:

  • You must not use libraries that are unsafe for threads. For instance, you should not use Motif or other GUI libraries that are not “multi-thread-safe.”

  • You should be careful if installing global event handlers, such as timers and signal handlers, if they override those that may already be installed by the host application. The safest policy is to avoid this altogether.

  • Where possible, you should avoid intentionally locking system resources such as physical memory blocks by using low-level UNIX calls or device drivers, because this can result in deadlock.

  • Your code should be reentrant. This means it should not rely on global state between calls, because it is possible for more than one instance of your converter to be running at the same time.

Converters are free to choose the size of the data blocks they read and write. GoldenGate writes into the buffer that your converter supplies during a SgCvtGetData() call. Your converter must allocate and free this buffer space as necessary. During a SgCvtSendData() call, your converter again supplies a buffer of data. The SgCvtSendData() call does not alter your buffer. If the call returns SG_CVT_E_SUCCESS to indicate that your data has been sent, or SG_CVT_E_FAILURE to indicate a general failure, free the buffer or re-use it as appropriate. If the call returns SG_CVT_E_AGAIN (you passed B_FALSE as the canwait argument) your data has not been sent, and you should retain it to try again later.

Example of a Simple Stream Converter

The following example shows a simple stream converter. It expects a stream of ASCII text characters, and outputs the stream with any uppercase characters replaced by their lowercase equivalents.

#include <SgCvt.h>
#include <dmedia/dm_params.h>

void CvtToLower
        (
        void *arg
        )
{
   SgCvtConverterData *data = (SgCvtConverterData *) arg;
   SgCvtStatus s;
   char buf[BUFSIZ];
   size_t nreq = BUFSIZ;
   unsigned int len=0;
   int start = 0;
   int i;

   /* Evaluation */
   if (data->operation == SG_CVT_REQ_EVALUATE) {

        /*
         * In less trivial converters, we would check for
         * valid params in the context, but in this case all
         * we're doing is byte translation, so we can always
         * say yes.
         */

        data->status_return = SG_CVT_E_ACCEPT;
        return;
   }

   /*
    * Conversion Loop.  A similar construct will appear in
    * all streaming converters.  The model is fetch data,
    * convert it and forward it, until we have forwarded the
    * end of stream, then jump out the loop.
    */
   for (;;) {

      s = SgCvtGetData(data->context, nreq, buf, &len, NULL, B_TRUE);
      if (s == SG_CVT_E_FAILURE) {
         fprintf(stderr, “converter: failed to get data\n”);
         return;
      }

      if (s == SG_CVT_E_END_OF_STREAM) {
        SgCvtSendEndOfStream(data->context);
        break;
      }

      /*** start converter-specific part ***/

      for (i=0; i<len; i++)
         buf[i] = tolower(buf[i]);

      /*** end converter-specific part ***/

      s = SgCvtSendData(data->context, (void *)buf,
                       len, NULL, B_TRUE);

      if (s == SG_CVT_E_FAILURE) {
         fprintf(stderr, “converter: failed to get data\n”);
         return;
      }
      start += (len);
   }
   /*
    * When we get here, this converter's work
    * is complete.  Others in the same pipeline may
    * still be running, but that's irrelevant to us.
    * We simply return.  If we were invoked in a dedicated
    * sproc “thread”, which is always the case for
    * streaming converters, this terminates it.
    */

   return;
}

Note the above comment about other converters: Others in the same pipeline may still be running. It is important to remember that your converter is almost always invoked as a subprocess of the application. “Programming Constraints” lists some considerations when writing converter code.

Building a DSO

GoldenGate converters reside in Dynamic Shared Object (DSO) libraries.

After you have written and tested your conversion function by calling it directly from a test program, you are ready package it as a GoldenGate converter.

This section covers the following topics:

Creating a DSO For Your Converter

Create a DSO for your converter. A simple Makefile (below) for the previous example, “Example of a Simple Stream Converter”, illustrates the compilation and linkage requirements for a GoldenGate DSO.

#
# Makefile for GoldenGate Sample Converter DSO
#
CVTR = CvtToLower
all:
     cc -c -I/usr/include/convert $(CVTR).c   
     ld -no_unresolved -o libUserCvtrs.so -shared $(CVTR).o

Creating a Converter Description File

After you compile your converter, you must create a converter description file that identifies your converter to GoldenGate. You use this file to test your converter, and intimately to register it with GoldenGate. A simple example for the CvtToLower converter follows.

#
# Lowercase Text Stream
#
Converter {
        Name:           “CvtToLower”
        IOMethod:       StreamToStream
        Input:          “MIXEDCASE”
        InputLabel:     “ASCII bytes, any case”
        Output:         “LOWERCASE”
        OutputLabel:    “ASCII bytes, lower case”
        Vendor:         “SGI (Sample)”
        Version:        “1.0”
        Description:    “Lowercases chars in input stream”
        DSO:            “/usr/people/fred/libFredsCvtrs.so”
        Function:       “CvtToLower”
}

Make sure the DSO field is set to the full pathname for the DSO you have built.

The grammar of the converter description file is fairly simple. Three types of statements exist; they are identified by the keywords Parameter, Converter, and Pipeline. Table G-4 defines the statements.

Table G-4. Converter Description File Statements

Statement

Description

Parameter statement

Defines a single parameter

Converter statement

Describes a converter and may include Parameter statements

Pipeline statement

Defines a series of converters to be used together, and may contain both Converter and Parameter statements

Some example descriptions follow. The easiest way to write a converter description file is to copy an existing one. You can use these examples, or copy entries from the default registry file, /var/GoldenGate/ConverterRegistry.

#
# Lowercase Text Stream
#
Converter {
        Name:           “CvtToLower”
        IOMethod:       StreamToStream
        Input:          “MIXEDCASE”
        InputLabel:     “ASCII bytes, any case”
        Output:         “LOWERCASE”
        OutputLabel:    “ASCII bytes, lower case”
        Vendor:         “SGI (Sample)”
        Version:        “1.0”
        Description:    “Lowercases chars in input stream”
        DSO:            “/usr/people/fred/libFredsCvtrs.so”
        Function:       “CvtToLower”
}

#
# Windows BMP to XWD
# 
Converter {
        Name:           “BMP_FILE_TO_XWD_FILE”
        IOMethod:       FileToFile
        Input:          “BMP_FILE”
        InputLabel:     “BMP_FILE”
        Output:         “XWD_FILE”
        OutputLabel:    “XWD_FILE”
        Vendor:         “SGI”
        Version:        “1.0”
        Description:    “BMP_FILE to XWD_FILE”
        DSO:            “libcvt_SGI.so”
        Function:       “xwdout”
}

#
# Windows BMP to Compuserv GIF-89, through JPEG (JFIF)
# This isn't necessary, since the default converters
# can go directly to GIF 89 from BMP, but it illustrates the # Pipeline syntax for chaining converters together.
Pipeline {
        Name:           “BMP_FILE_TO_GIF_89_FILE”
        IOMethod:       FileToFile
        Input:          “BMP_FILE”
        InputLabel:     “Windows BMP”
        Output:         “GIF_89_FILE”
        OutputLabel:    “Compuserve GIF”
        Vendor:         “SGI”
        Version:        “1.0”
        Description:    “Windows BMP to GIF, via JPG”
   Converter {
        Name:           “BMP_FILE_TO_JFIF_FILE”
        IOMethod:       FileToFile
        Input:          “BMP_FILE”
        InputLabel:     “BMP_FILE”
        Output:         “JFIF_FILE”
        OutputLabel:    “JFIF_FILE”
        Vendor:         “SGI”
        Version:        “1.0”
        Description:    “BMP_FILE to JFIF_FILE”
        DSO:            “libcvt_SGI.so”
        Function:       “jfifout”
   }
   Converter {
        Name:           “JFIF_FILE_TO_GIF_89_FILE”
        IOMethod:       FileToFile
        Input:          “JFIF_FILE”
        InputLabel:     “JFIF_FILE”
        Output:         “GIF_89_FILE”
        OutputLabel:    “GIF_89_FILE”
        Vendor:         “SGI”
        Version:        “1.0”
        Description:    “BMP_FILE to GIF_89_FILE”
        DSO:            “libcvt_SGI.so”
        Function:       “gifout”
   }
}

Testing Your Converter

To test your converter, first verify that your converter description file is valid and does not cause the GoldenGate built-in registry parser to fail.

Set the environment variable CVT_REGISTRY_OVERRIDE to the full pathname of the converter description file you just created:

setenv CVT_REGISTRY_OVERRIDE /usr/people/fred/my_registry.cdf

Then run a test program that will exercise the parser. The gg_query demo program that comes with GoldenGate is good for this. Find it in /usr/share/src/GoldenGate (if you haven't already done so, install the demo programs from your IRIX distribution media). Copy the demo programs to your own directory, go into the Query subdirectory, and type make. Then execute the gg_query program. The output should look like this:

Converter (CvtToLower):
        method: Stream To Stream
        input:  MIXEDCASE (ASCII bytes, any case)
        output: LOWERCASE (ASCII bytes, lower case)
        vendor: SGI (Sample)
        version:        1.0
        descr.: Lowercases chars in input stream
        DSOname:        /usr/people/fred/libFredsCvtrs.so
        Function:       CvtToLower

If you see an error message, go back and check that your converter description file is valid, checking especially that all string values are properly quoted. Also check that the GoldenGate software is properly installed by unsetting the CVT_REGISTRY_OVERRIDE variable and re-executing the gg_query program. It should list the default converters installed on the system (over 100 of these exist).

Once the test runs successfully, you are ready to try executing your converter. You can use your own program, or the demo programs in the ConvertFile and ConvertStream directories to do this. Each program prints a help message describing its arguments if you run it with no arguments.

After you are satisfied that your converter works when executed via GoldenGate, you are ready to make it available to other applications on the system. Unset the CVT_REGISTRY_OVERRIDE variable; you are finished unit-testing your converter.

Registering Your Converter

To register your converter, you must add your converter description file to the system registry.

The System Registry

The system registry is a text file that uses the same syntax as your converter description file. Just edit the file /var/GoldenGate/ConverterRegistry (you must be a privileged user to do this) and add your entry wherever you like.

Look at the attributes of the converters already registered. If there are potential clashes with your converter, you may wish to insert your converter closer to the beginning of the registry. Some applications may decide to convert using the first converter they find that appears to satisfy their requirements, rather than evaluating the alternatives. If you want to make sure this kind of application executes your converter rather than another one that could do that same conversion, insert your entry closer to the beginning of the file.

Some Registry Syntax Details

The most important fields are those that the service uses to locate the executable converter module: the DSO name and the Function name. The other fields are primarily for display by administration tools, and for applications to query the registry. The Input and Output fields are strings that must exactly match the format names that applications will use to search for converters. For instance, where there are naming conventions such as ICCCM target names, these should be used exactly.

See “Supported Target Formats” in Chapter 7 for a list of standard input and output formats supported by the default converters supplied with GoldenGate.

Parameters can be one of two types: Constraint or Programmable.

Constraint parameters are used to specify constant values for a data attribute in the description file. When you see a constraint parameter, it means that this converter always sets the corresponding data attribute to the stored value, overriding its current value in the input.

Programmable parameters are used to specify parameters that are set at runtime based upon the requested input and output parameters. Programmable parameters are used to pass a runtime parameter to one of the stages of a pipeline. For example, if you have a two-stage pipeline designed to scale an SGI image to an arbitrary size, then convert it to JPEG, you want to pass one of the output parameters (the required output size) to the first stage of the converter. You do this by specifying a programmable parameter for the first stage.

Installing Your Converter

Make sure your installation copies the DSO containing your converter to the standard location for converter DSOs: /usr/lib/convert. If you install your library there, you can use a relative DSO name in your converter description file. If you install anywhere else, you must use a full path name in the registry to ensure that the service will find your converter.

See the GoldenGate Release Notes (type relnotes goldengate) for information about installation.

Some Sample Converters

This section presents annotated sample code for two different converters:

A Simple StreamToStream Converter - UpperCase

This type of converter can often offer the best performance in many circumstances, because all the knowledge of the conversion operation is in the converter itself, and because it typically does not need to access the filesystem to achieve conversion. It is appropriate when the data format is naturally streamable, such as ASCII text or other self-identifying or raw data.

The converter used in this example performs a simple mapping of mixed-case text to uppercase text. The converter-specific parts are clearly marked. These are the lines that you will replace with your own task-specific conversion code. The remainder is boilerplate code that can be re-used in many different converters.

/* converter function */
void UpperCase
(
    void *arg
)
{
    SgCvtConverterData *data = (SgCvtConverterData *) arg;
    SgCvtStatus s;

The next 2 lines are somewhat task-specific. Your converter should use a buffer size appropriate to the data type and the task. Careful selection of a buffer size will yield better performance in many cases. For instance, if your converters needs to operate on audio or movie “frames,” then you may choose to read and write buffers that represent whole numbers of frames.

    char buf[BUFSIZ];
    size_t nreq = BUFSIZ;

    unsigned int len=0;
    int start = 0;
    int i;

The next part is the Evaluation section. Our converter operates on a byte stream: if a byte represents a lower-case character in the current locale, we are going to uppercase it. Otherwise it passes through untouched. It is appropriate for this converter to accept any stream; it does not need to evaluate parameters.

    /* Evaluation */
    if (data->operation == SG_CVT_REQ_EVALUATE) {
        data->status_return = SG_CVT_E_ACCEPT;
        return;
    }

The following loop does the conversion, one block at a time. The loop terminates when the end of stream is detected.

    /* Conversion */
    for (;;) {

        s = SgCvtGetData(data->context, nreq, buf, &len,
                         NULL, B_TRUE);

        if (s == SG_CVT_E_END_OF_STREAM) {
            SgCvtSendEndOfStream(data->context);
            break;
        }

These two lines show the entire task-specific code requirements for the uppercase text converter. Your converter will substitute its own conversion-specific code for these lines. The model is the same in each case; the converter generates a buffer to be sent from the buffer it has received, by applying a specific conversion algorithm.

        for (i=0; i<len; i++)
            buf[i] = toupper(buf[i]);

When the new buffer has been generated, your converter sends it into the pipeline. In this example, the data was converted in place. Sometimes that is not possible, because the converted data will not fit in the original buffer. In these cases, your converter may allocate, populate, send, then free a dynamic buffer each time through the loop.

        s = SgCvtSendData(data->context, (void *)buf, len,
                          NULL, B_TRUE);

        start += (len);
    }

After sending all the converted data, and calling SgCvtSendEndOfStream, your converter can return. You should always set the status_return field.

    data->status_return = SG_CVT_E_SUCCESS;
    return;
}

A FileToFile Converter - UNIX Man Page File to HTML File

Often, to convert data from one application into a form usable by another, you need to save the data to a file in one format, convert it using an IRIX command-line translator program, then open the new file using the application you want to use.

Applications using components that are integrated with GoldenGate can eliminate the intermediate end-user steps. The same external translator command can be packaged as a GoldenGate converter and invoked automatically on behalf of the user.

It is quite straightforward to integrate an existing IRIX command with GoldenGate. Your main task is to write a function that the service can invoke, which constructs from its parameters a command line for the translator program.

The converter described here provides GoldenGate access to a command named man2html, which converts troff source files for UNIX man pages into HTML files that can be viewed using a Web browser. It can be used by a CGI script that implements an online help system for remote users running Web browsers.

The command itself takes one argument: the input file name. It writes its output to stdout. The job of our converter is to make this interface look like any other GoldenGate converter.

Converter functions always require the SgCvt.h header file and always have the have the same calling convention:

#include <libgen.h>
#include <SgCvt.h>
void ManToHtml(void *arg)
{

First, cast the data argument to the right type.

    SgCvtConverterData *data = (SgCvtConverterData *) arg;

Then, define some other local variables. Most of these are the same in every converter of this type that you write.

    SgCvtStatus s;
    SgCvtContextInfo ctx_info;
    char cmd[BUFSIZ];
    int sys_status = 0;
    char *cmdpath;

Next, handle converter evaluation requests. This converter is very simplistic: there are no parameters, and it just ACCEPTs the request. In real converters, always provide proper evaluation of any parameters, especially if it is expensive for the application to try the conversion and fail.

    /* Evaluation */
    if (data->operation == SG_CVT_REQ_EVALUATE) {
        data->status_return = SG_CVT_E_ACCEPT;
        return;
    }

The remaining code handles conversion requests. Note that it is never reached unless the caller requests conversion because the evaluate section has its own return statement.

The first thing to do is check that the program you are wrapping is installed and that you can execute it.

    /* conversion */
    cmdpath = pathfind(getenv (“PATH”), “man2html”, “rx”);
    if (cmdpath == NULL) {
        data->status_return = SG_CVT_E_MISSING_COMMAND;
        return;
    }

Then, extract the input and output filenames from the conversion context.

    (void) SgCvtGetContextInfo(data->context,
        SG_CVT_INFO_INPUT_FILE   |
        SG_CVT_INFO_OUTPUT_FILE,
        &ctx_info);

At this point you know everything needed to construct the command you are going to execute. You use the UNIX system(2) call to execute the conversion, so the next step is to create the command line.

    sprintf(cmd, “%s %s > %s 2> /dev/null”, cmdpath,
         ctx_info.input_file, ctx_info.output_file);

Finally, execute the command, and set the return status to indicate whether it worked before returning.

    sys_status = system(cmd);
    data->status_return =
        sys_status ? SG_CVT_E_FAILURE : SG_CVT_E_SUCCESS;

    return;
}

Note that your converter is normally run as a subprocess of the invoking application. You should not call exit(2) to terminate your converter; you should simply return, allowing GoldenGate and the operating system to take care of managing conversion threads.