The Electronic Light Table (ELT) operator implements a chain of operators in hardware. This implementation allows you to view and manipulate image files, compressed or not, in real time. ELT also allows you to add graphics and text on top of manipulated images.
Viewing the processed images in real time avoids the necessity of first storing them first on disk before viewing it. Most often you use an ELT application to look at huge images (10K X 10K or larger).
This chapter describes the ilELTImg operator and how you use it along with ilDisplay, ilView, and ilStereoView to create an ELT application. This chapter contains the following major sections:
You can think of the ELT as an image-processing pipeline, as shown in Figure G-1.
Each stage of the ELT pipeline has the following purpose:
| R Sets | contain the image data at different levels of reduction. | |
| DeWarp | changes the perspective or corrects any image imperfection caused by the image capturing mechanism. | |
| RotZoom | allows you to zoom or rotate the image. ELT selects the correct R-set to input for each specified zoom value. | |
| Convolve | allows you to sharpen or blur the image. | |
| Histogram | feeds data into the LUT to adjust pixel luminance dynamically according to the overall brightness of the displayed image. | |
| LUT | adjusts pixel luminance and color dynamically. |
The output pages are subdivided into fine meshes of a user-defined size. The default size is a uniform, 10 X 10 mesh. The dewarp function composited with an affine transformation is used to compute the coordinates of the input image. Using a fine mesh with a high order resampling method, either bilinear or bicubic interpolation, creates high quality dewarped images.
The member functions used to get and set both dewarp and mesh parameters are
ilStatus setWarp(ilELTrset* rset, const ilWarp* Xyc); ilStatus getWarp(ilELTrset* rset, const ilWarp*& Xyc) const; void setMaxMeshSize(int n); int getMaxMeshSize() const; ilStatus setResampType(ilResampType rs); |
rset allows the application to reset the warp operator, Xyc is the dewarp function, n is the maximum mesh size, and rs is either ilNearNb, ilBiLinear or ilBiCubic.
The RotZooming operator enables fluid zoom, rotation, and translation of images. You use the following ilELTImg member functions to control the view manipulation:
void setAngle(float ang); float getAngle() const; void setZoom(float s); float getZoom() const; void setHorizontalFlip(int flip); int getHorizontalFlip(); void setVerticalFlip(int flip); int getVerticalFlip(); } |
where ang is the angle to display the image at, myELT is the ilELTImg object, s is the zoom level, and flip indicates that the image should be flipped.
The ilELTImg class also provides member functions for convolving an image using either general or separable kernels.
void setConvKernel(ilKernel* inputKernel); const ilKernel* getConvKernel() const; |
where inputKernel is the kind of convolution kernel you want to set. The kernel may be either an ilKernel- or ilSepKernel-type of object.
Kernels with sizes of 3 X 3, 5 X 5, or 7 X 7 can be accelerated on the Impact, RealityEngine, and InfiniteReality platforms. Separable convolutions run faster than general convolutions.
Histogram data is used by the look-up table operator, iflLutImg, to automatically adjust the color and luminance of each pixel to spread the range of luminance in an image over the number of bits per pixel in the output image. Transposing the luminance values in this way has the effect of making dark images brighter and overly-bright images darker.
Histogram data is collected over a user-definable number of frames. If, for example, the sampling interval is defined as thirty frames, data is collected over the first 28 frames, during the display of the 29th frame the data is read out of the hardware, and during the 30th frame the information is sent to the LUT. In general, data is collected over the sampling interval minus two frames (samplingInterval - 2).
Collecting and processing histogram data incurs a performance drawback. To alleviate this problem, you can specify the histogram sampling interval. Performance improves if the histogram data is sampled infrequently. The possible consequence of a long sampling interval, however, is sudden adjustments to luminance and color. The default update interval is once per 30 frames.
Use the following ilELTImg member functions to control the sampling rate:
ilStatus setHistPeriod(int numFrames); int getHistPeriod() const; |
where numFrames is the number of frames that pass between one sampling and another.
Use the following functions to specify the number of bins used to compute the histogram tables:
ilStatus setHistNbins(int num); int getHistNbins() const; |
where num is the number of bins.
All channels have the same number of bins and the number of bins must be a power of two.
Use the following functions to process the histogram tables that are returned:
ilStatus setHistCallback(const ilCallback* callback = NULL); ilCallback* getHistCallback() const; |
The look-up table (LUT) adjusts the luminance and color values of each pixel dynamically. The LUT can adjust the luminance and color values on a frame-by-frame basis. The actual adjustment is determined by how often the histogram samples the image. The default is once every 30 frames.
The process of Dynamic Range Adjustment (DRA) takes the darkest and brightest luminance values in the image and scales them over the number of bits per pixel in the output image. This has the effect of making dark images bright and overly-bright images darker.
The Tonal Transfer Characteristic (TTC) changes the image colors. These changes can either correct color errors introduced by a camera mechanism or create false colors for the purpose of examining image details.
The member functions used to get and set DRA and TTC are
ilStatus setLookUpTable(const iflLut& lut); const iflLut* getLookUpTable() const; |
where lut contains information from the lookup table.
Before displaying the image manipulated by the ELT chain, you can add vectors, shaded surfaces, or text on top of the images by using the callback functions provided by ilELTImg and ilDisplay. Four callbacks are provided in the ilELTImg class for
adjusting the contrast
computing the bounding box
generating triangle meshes
deleting buffer resources
The callbacks that adjust the contrast are discussed in “Collecting Histogram Data”. The other callback functions include
void setBBoxCallback(const ilCallback* callback = NULL); ilCallback* getBBoxCallback() const; void setTmeshCallback(const ilCallback* callback = NULL); ilCallback* getTmeshCallback() const; void setTmeshDelCallback(const ilCallback* callback = NULL); ilCallback* getTmeshDelCallback() const; |
The bounding box callback returns the vertices of the input space. If NULL is passed into the setBBoxCallback() function, the default bounding box calculations method is used.
The mesh callbacks generate the triangle mesh necessary to dewarp an image. If NULL is passed into the setTmeshCallback() function, the default Tmesh calculations method is used.
You can either delete allocated memory for the Tmesh explicitly using setTmeshDelCallback(), or let it deallocate itself automatically when all processing associated with the callback is complete.
You can enable or disable most of the operators in the ELT chain using the following functions:
void enableHistogram(int enable = TRUE); void enableConv(int enable=TRUE); void enableLut(int enable=TRUE); |
You can make sure the operators are enabled using the following functions:
void isHistEnabled(); void isConvEnabled(); void isLutEnabled(); |
In addition to enabling and disabling operators, you can set their values.
Before you enable any of the operators in an ELT chain, you must specify parameters for them. For several of the operators, you can only set their values by using ilELTImg member functions, such as setConv(). Other operator values can be set using the ilELTImg constructor.
ilELTImg(ilImage* img=NULL, float minZoom=0.5, float maxZoom=8.0,ilWarp_2d* warp=NULL, int isR0=TRUE, void* rsetInfo=NULL); |
In the constructor, img represents the input image; generally, this is the R0 image. minZoom and maxZoom specify the minimum and maximum values of the zoom range.These parameters often correspond to the range of images in the R-set. The warp argument specifies the two-dimensional maps used to dewarp the image from the 1.0X plane coordinates to the R-set's image plane coordinates. The isR0 argument specifies whether or not the image is the R0 image. Finally, the rsetInfo is a pointer to user-specific R-set image data. ilELTImg's member function, getRsetInfo() can use this information.
The ELT takes advantage of three processes to accelerate image manipulation processing:
look-ahead algorithms
hardware implementation of operator functions
reprocess the image only when there is a change in operator value
This section looks at each of these processes.
ELT implements look-ahead algorithms that accelerate the roaming and zooming operations carried out in the ELT chain. The ELT uses one extra page border surrounding the display window for look-ahead processing. Internally, texture memory, used for intermediate buffers, is allocated in powers of two. For example, if a 1K by 1K space is allocated as the intermediate, internal buffer, the maximum windows size is 896 by 896 because the look-ahead page border takes up 128 pixels (2 X 64) in each dimension.
You can enable or disable the look-ahead mechanisms using the following functions:
void enableRoamLookAhead(int enable = TRUE); void enableZoomLookAhead(int enable = FALSE); |
By default, the roaming look-ahead mechanism is enabled and the zooming look-ahead mechanism is disabled. The default values are appropriate if the user will roam but not zoom. If the user will zoom without roaming, it is better to enable the zoom look-ahead and disable the roam look-ahead mechanism. If the user will zoom and roam, both look-ahead mechanisms should be enabled. In this case, however, the performance of one look-ahead mechanism may suffer because of the processing of the other look-ahead mechanism.
You can make sure the look-ahead algorithm is enabled by using the following functions:
void isRoamLookAheadEnabled(); void isZoomLookAheadEnabled(); |
You can use an ELT application to view and manipulate an image in real time partly because the manipulation processing is carried out in specialized hardware. You can turn off this functionality using the following function, however, the performance of your ELT application will be severely impacted.
void enableFastPath(int enable=TRUE) {fastPath=enable; setAltered();}
|
You can make sure the hardware implementation is enabled by using the following function:
int isFastPathEnabled(); |
Although there is no size limit for the input image, images are stored and processed internally in 64 X 64 tiles.
Images are most often saved in multiple magnifications. Together, all the images at different magnifications are called a Reduced Resolution Data Set, or R-set. If the original image is named R0, R1 is generally R0 minimized two times; R2 is R1 minimized two times, and so on. Each of these minimizations can be filtered for optimal results. When the user zooms from one magnification to another, the ELT operator actually chooses the appropriate R-set to input to the ELT pipeline. R-sets often contain seven different magnifications of the original image.
The output of the ilELTImg operator can be attached to an ilView object to display a single image, multiple images, or a stereo display. Single and multiple image displays are implemented using ilDisplay and ilView objects. Stereo views are implemented using ilStereoView, a derived class of ilView, on RealityEngine and InfiniteReality platforms.
ilStereoView renders left images to the left buffer and right images to the right buffer. The images are displayed alternately to create a three-dimensional image when viewed through special glasses.
The software distribution includes a full-blown example of an ELT application called ilChain in /usr/share. This section presents a simplified example of an ELT application. It is based on the example code, ilrzview.c++, found in /usr/share.
This example implements the following user interface:
Dragging with the left mouse button moves the view in the display.
Dragging with the center mouse button moves the image in the view.
Using the left and right arrows on the keyboard rotate the image.
Using the up and down arrows on the keyboard zoom the image.
Example G-1 uses the following steps to implement an ELT application:
Parse the command line arguments.
Open an image file.
Instantiate an ilELTImg object.
Create an X connection and open a display.
Create an X window viewer.
Implement the user interface.
#include <stdlib.h> #include <string.h> #include <getopt.h> #include <stdio.h> #include <X11/Xlib.h> #include <X11/keysym.h> #include <il/ilFileImg.h> #include <il/ilViewer.h> #include <il/ilConfigure.h> #include <ifl/iflMinMax.h> #include <il/ilELTImg.h> #include <il/ilBlurImg.h> void main (int argc, char* argv[]) { // Step 1: Process the command line arguments int usage = 0, sizePres = FALSE, attr = 0, autoAbort = FALSE; int compSize = 8; int blur = FALSE; iflSize winsize; ilResampType resamp = ilBiLinear; int c; while ((c = getopt(argc, argv, “bnlcads:”)) != -1) { switch (c) { case `b': blur = TRUE; break; case `n': resamp = ilNearNb; break; case `l': resamp = ilBiLinear; break; case `c': resamp = ilBiCubic; break; case `a': autoAbort = TRUE; break; case `d': compSize = 4; attr |= ilVisDoubleBuffer; break; case `s': sizePres = TRUE; sscanf(optarg, “%d,%d”, &winsize.x, &winsize.y); break; case `?': usage = 1; break; } } if (usage || argc-optind != 1) { printf(“autoAbort, doubleBuffer, Size\n”); printf(“%s [-ad -s <size x,y>] <image-file>\n”, argv[0]); exit(0); } // Step 2: Open an image file. ilFileImg img(argv[optind]); if (img.getStatus() != ilOKAY) { char buf[400]; rintf(stderr, “Couldn't open image file %s: %s\n”, argv[optind], ilStatusToString(img.getStatus(), buf, sizeof(buf))); exit(0); } // Step 3: Instantiate an ilELTImg object. ilELTImg myELT(image, .5, 8)); // Step 4: Get an X connection and open a display. if (!sizePres) img.getSize(winsize, iflLowerLeftOrigin); ilImage* image = &img; if (blur) image = new ilBlurImg(image); // Get display connection and clamp window size Display* dpy = XOpenDisplay(NULL); int screen = DefaultScreen(dpy); winsize.x = iflMin(winsize.x, DisplayWidth(dpy, screen)); winsize.y = iflMin(winsize.y, DisplayHeight(dpy, screen)); // Step 5: Create an X window viewer. ilViewer viewer(dpy, winsize.x, winsize.y, attr, compSize); if (autoAbort) { viewer.enableQueueing(); viewer.enableAutoAbort(); } ilView* view = viewer.addView(&myELT, ilClip|ilCenter); view->setAutoCenter(); // Step 6: Implement the user interface. int done=FALSE; float zoom=1; int angle=0; int movieRunning = FALSE; while (!done) { XEvent e; if (movieRunning) { if (!XCheckWindowEvent(dpy, viewer.getWindow(), -1, &e)) { int z = view->getZ() + 1; view->setZ(z); viewer.paint(); continue; } } else XNextEvent(dpy, &e); switch (e.type) { case KeyPress: switch(XLookupKeysym(&e.xkey, 0)) { // center the view in the viewer case XK_Home: viewer.display(NULL, ilCenter|ilClip); break; // control-Q and escape exit the program case XK_q: if (!(e.xkey.state&ControlMask)) break; /*FALLTHROUGH*/ case XK_Escape: done = TRUE; break; case XK_s: movieRunning = FALSE; break; // flip the image case XK_h: viewer.abort(); myELT.setHorizontalFlip(!myELT.getHorizontalFlip()); viewer.paint(); break; case XK_v: viewer.abort(); myELT.setVerticalFlip(!myELT.getVerticalFlip()); viewer.paint(); break; // zoom and rotate the image case XK_Up: viewer.abort(); myELT.setZoom(zoom *= 1.2); viewer.paint(); break; case XK_Down: viewer.abort(); myELT.setZoom(zoom /= 1.2); viewer.paint(); break; case XK_Right: viewer.abort(); myELT.setAngle(angle -= 15); viewer.paint(); break; case XK_Left: viewer.abort(); myELT.setAngle(angle += 15); viewer.paint(); break; } break; case DestroyNotify: viewer.destroyNotify(); done = TRUE; break; default: viewer.event(&e); break; } } }
The ilELTImg class has an extensive set of methods. The ilELTImg manpage contains an extended discussion of each method. Table G-1 summarizes each method.
Table G-1. Methods in ilELTImg
Method | Description |
|---|---|
ilELTImg | ilELTImg(ilImage* img=NULL, float minZoom = 0.5, Constructor for the class. |
addRset | ilELTrset* addRset(ilImage* img, float minZoom, Allows the user to specify additional R-sets for roaming. |
enableConv | void enableConv(int enable=TRUE) Enables or disables the convolution operation on an |
enableFastPath | void enableFastPath(int enable=TRUE) Enables or disables the special-purpose hardware acceleration for ELT. |
enableHistogram | void enableHistogram(int enable = TRUE) Enables or disables the auto histogram operation on an ilELTImg object. |
enableLut | void enableLut(int enable=TRUE) Enables or disables the table look-up operation on an ilELTImg object. |
enableRoamLookAhead | void enableRoamLookAhead(int enable = TRUE) Enables or disables the roaming look-ahead operation on an ilELTImg object. |
enableRset | ilStatus enableRset(ilELTrset* rset, int enable = TRUE) Enables or disables a selected R-set to be in effect. |
enableZoomLookAhead | void enableZoomLookAhead(int enable = TRUE) Enables or disables the zooming look-ahead operation on an ilELTImg object. |
mapFromInput | void mapFromInput(float& u, float& v, float& w, Given a point (x, y, z) in the R0 image plane (regardless of which R-set is currently being roamed), compute (u, v, w) in the display plane using the mapping specified for geometric transformation. |
mapToInput | void mapToInput(float& x, float& y, float& z, float u, Given a point (u, v, w) in the display plane, compute (x, y, z) in the R0 image plane (regardless of which R-set is currently being roamed) using the mapping specified for geometric transformation (including both the dewarp and the affine-transformation functions). |
getAngle | float getAngle() const Returns the angle of rotation (in degrees) specified for the view. |
getBBoxCallback | ilCallback* getBBoxCallback() const Returns the user-provided callback which is used to compute the bounding box of a given output page mapped in its input image space. |
getBicubicFamily | void getBicubicFamily(float& b, float& c) const Returns the B and C terms, defining the cubic resampling coefficients, in b and c. |
getConvBias | double getConvBias() const Returns the current additive bias specified for convolution as a double. |
getConvKernel | const ilKernel* getConvKernel() const Returns the current convolution kernel. |
getEnabled | int getEnabled() The returned value is comprised of one or more of the following bit fields: ilELTImg::ilEPdewarp is set if dewarp operation is enabled. ilELTImg::ilEPhist is set if auto histogram operation is enabled. ilELTImg::ilEPconv is set if convolution is enabled. ilELTImg::ilEPlut is set if table look-up is enabled. ilELTImg::ilEProam is set if roaming look-ahead is enabled. ilELTImg::ilEPzoom is set if zooming look-ahead is enabled. |
getHistNbins | int getHistNbins() const Returns the number of bins currently being specified for the histogram table. |
getHistPeriod | int getHistPeriod() const Returns the number of frames between look-up table updates; default is 30 frames. |
getHistCallback | ilCallback* getHistCallback() const Returns the current histogram callback being used or NULL if none has been defined. |
getLookUpTable | const iflLut* getLookUpTable() const Returns the current lookup table being provided by the user. |
getMaxMeshSize | int getMaxMeshSize() const Returns the maximum allowable mesh size used for geometric transformations. |
getPairedImg | ilELTImg* getPairedImg() const Returns the other image of a paired image, for example, in a stereo image. |
getResampType | ilResampType getResampType() const Returns the current resampling type used in geometric transformations. |
getRset | ilELTrset* getRset(ilImage* img) const Returns an opaque handle to the R-set for the input image, img. |
getRsetChangeCallback | ilCallback* getRsetChangeCallback() const Returns the user-provided callback which indicates when the input image to the ELT has switched from one R-set to another. |
getRsetInfo | void* getRsetInfo(ilELTrset* rset) const Returns the user-provided information about the specified R-set. If no R-set information has been provided, NULL is returned. |
getRsetZoomRange | ilStatus getRsetZoomRange(ilELTrset* rset, Gets the scaling range covered by the specified R-set. |
getTmeshCallback | ilCallback* getTmeshCallback() const Returns the user-provided callback which a generates triangle mesh for a given output page. |
getWarp | ilStatus getWarp(ilELTrset* rset, const ilWarp*& func) const Returns rset's dewarp function in func. |
getZoom | float getZoom() const Returns the current display scale being specified. |
isConvEnabled | int isConvEnabled() Returns TRUE if convolution is enabled for the current operation; FALSE, otherwise. |
isFastPathEnabled | int isFastPathEnabled() Returns TRUE if special purpose hardware acceleration is enabled for the current operation; FALSE, otherwise. |
isHistEnabled | int isHistEnabled() Returns TRUE if auto histogram is enabled for the current operation; FALSE, otherwise. |
isLutEnabled | int isLutEnabled() Returns TRUE if table look-up is enabled for the current operation; FALSE, otherwise. |
isRoamLookAheadEnabled | int isRoamLookAheadEnabled() Returns TRUE if roaming look-ahead is enabled for the current operation; FALSE, otherwise. |
isZoomLookAheadEnabled | int isZoomLookAheadEnabled() Returns TRUE if zooming look-ahead is enabled for the current operation; FALSE, otherwise. |
removeRset | ilStatus removeRset(ilELTrset* rset) Removes the specified R-set from the roaming operation. |
setAngle | void setAngle(float ang) Changes the rotation angle to that specified by the argument, ang, which is specified in degrees. |
setBBoxCallback | void setBBoxCallback(ilCallback* callback = NULL) Sets up a callback to compute the bounding box of a given output page mapped in its input image space. |
setBicubicFamily | void setBicubicFamily(float b=1., float c=0.) Specifies the B and C terms which define the bicubic resampling coefficients. |
setConvBias | ilStatus setConvBias(double biasVal) Sets the additive bias applied to all pixels after the convolution operation. |
setConvKernel | void setConvKernel(ilKernel* inputKernel, Sets the convolution kernel. |
setHistNbins | ilStatus setHistNbins(int num) Sets the number of bins in the histogram table. |
setHistPeriod | ilStatus setHistPeriod(int numFrames) Sets the number of frames between LUT updates. |
setHistCallback | ilStatus setHistCallback(ilCallback* callback = Provides a histogram callback. |
setLookUpTable | ilStatus setLookUpTable(const iflLut& lut) Sets a new look-up table to be downloaded to the graphics system. |
setMaxMeshSize | void setMaxMeshSize(int N) Sets the maximum allowable mesh size for geometric transformation to be NxN pixels. |
setPairedImg | void setPairedImg(ilELTImg* pairedImg) Sets the other image of an image pair (i.e., stereo pair). |
setResampType | ilStatus setResampType(ilResampType rs) Selects the resampling type to be used. |
setRsetChangeCallback | void setRsetChangeCallback(ilCallback* callback = Provides a R-set change callback. |
setRsetInfo | ilStatus setRsetInfo(ilELTrset* rset, void* rsetInfo = Sets the user-specified R-set information for the given R-set handle. |
setRsetZoomRange | ilStatus setRsetZoomRange(ilELTrset* rset, Sets the scaling range covered by the specified R-set. |
setTmeshCallback | void setTmeshCallback(ilCallback* callback = NULL) Provides a triangle mesh generation callback. |
setTmeshDelCallback | void setTmeshCallback(ilCallback* callback = NULL) Sets up a callback to clean up triangle lists after the system finishes drawing them. |
setWarp | ilStatus setWarp(ilELTrset* rset, const ilWarp* func) Sets rset's dewarp function to be func. |
setZoom | void setZoom(float scale) Changes the display scale to scale. |