Technical Note TN2044

Decompressing DV frames and accessing the pixels

This technical note discusses using Decompression Sequences to decompress DV frames to an RGB offscreen destination, for the purpose of accessing each frames pixels directly. The same steps can be applied to any QuickTime supported compressed image format represented by an Image Description. The compressed image data can be sourced from a QuickTime Movie, the Sequence Grabber or from an application supplied buffer.

Introduction Getting Started The Image Description The Offscreen Setting up the Sequence Decompressing a Frame Asynchronous Decompression Accessing the Pixels Ending the Sequence References Document Revision History

Introduction

The overall procedure of decompression single DV frames from a buffer, to an RGB offscreen, then accessing the pixels directly is straightforward, especially if you don't care about scheduled decompression. However, there are a couple of different ways to go about it. You could for example, treat each frame as a single image, and use DecompressImage or FDecompressImage to decompress the "single-frame" into a pixel map. While this is fine for a single image, it is not the fastest way to decompress a series of images of the same format, sharing a common image description represented by the ImageDescription structure.

A more efficient method is the use of a Decompression Sequence.

The Image Compression Manager provides a set of APIs allowing developers to decompress sequences of images sharing a common image description. Each image in the sequence is referred to as a frame. A decompression sequence is started by calling DecompressSequenceBeginS, and each frame in the sequence is processed by calling DecompressSequenceFrameWhen. Upon completion, the sequence is closed with a call to CDSequenceEnd. Checking the status of the current operation can be done by calling CDSequenceBusy. There are also a number of DSequence Parameter APIs allowing the manipulation of the parameters controlling the decompression sequence, these include the source rectangle, transformation matrix and accuracy to name a few.

Getting Started

Determine where to decompress the image, build an image description describing the source data, decide how much of the image to decompress, and build a mapping matrix for the operation. These parameters must be specified in the call to DecompressSequenceBeginS when making a decompression request.

The destination is specified as a graphics port and the image source size is described by a rectangle, in the coordinate system of the source image. NULL can be used to specify the entire source image. To support transformations such as scaling to a destination of a different size, specify a matrix describing how the image is to be mapped into the destination graphics port.

You can figure out the size of the source image by examining the image description structure associated with the image or if you are intimately familiar with the image data you can create the image description yourself. The image description structure contains information defining the characteristics of the compressed image or sequence. One image description structure may be associated with one or more compressed frames. See Listing 1.

For a detailed discussion regarding the representation of uncompressed Y´CbCr video in an Image Description structure, including Image Description Extensions refer to Ice Floe #19.

Back to Top

The Image Description

Create an Image Description for your compressed data. See Listing 2.

Listing 1: Image Description structure.

struct ImageDescription {
    long idSize; // total size of ImageDescription including extra data
                 // (CLUTs and other per sequence data)
    CodecType cType; // what kind of codec compressed this data
    long      resvd1; // reserved for Apple use
    short     resvd2; // reserved for Apple use
    short     dataRefIndex; // set to zero
    short     version; // which version is this data
    short     revisionLevel; // what version of that codec did this
    long      vendor; // whose codec compressed this data
    CodecQ    temporalQuality; // what was the temporal quality factor
    CodecQ    spatialQuality; // what was the spatial quality factor
    short     width; // how many pixels wide is this data
    short     height; // how many pixels high is this data
    Fixed     hRes; // horizontal resolution
    Fixed     vRes; // vertical resolution
    long      dataSize; // if known, the size of data for this
                               // image descriptor
    short     frameCount; // number of frames this description
                               // applies to
    Str31     name; // name of codec ( in case not installed )
    short     depth; // what depth is this data (1-32) or
                               // (33-40 grayscale)
    short     clutID; // clut id or if 0 clut follows
                               // or -1 if no clut
};

Listing 2: Create an Image Description describing the compressed data.

// Create an Image Description describing the compressed data,
// you are responsible for disposing of the returned handle.
// Modify the Image description values to deal with any other
// compressed data formats - for example PAL (kDVCPALCodecType)
ImageDescriptionHandle MakeImageDescriptionForNTSCDV(void)
{
    ImageDescriptionHandle hImageDescription = NULL;

    hImageDescription =
        (ImageDescriptionHandle)
        NewHandleClear(sizeof(ImageDescription));

    if (NULL != hImageDescription) {
        (**hImageDescription).idSize = sizeof(ImageDescription);
        (**hImageDescription).cType = kDVCNTSCCodecType;

                // DV has no temporalQuality
        (**hImageDescription).temporalQuality = 0;
        (**hImageDescription).spatialQuality = codecNormalQuality;
        (**hImageDescription).width = 720;
        (**hImageDescription).height = 480;
        (**hImageDescription).hRes = 72 << 16;
        (**hImageDescription).vRes = 72 << 16;
        (**hImageDescription).frameCount = 1;
        (**hImageDescription).depth = 24;
        (**hImageDescription).clutID = -1;
    }

    return hImageDescription;
}

Back to Top

The Offscreen

Use QTNewGWorld to create an offscreen to decompress into. See Listing 3. Developers should note 24-bit (k24RGBPixelFormat) is not the best destination for the DV Codec, 32-bit (k32ARGBPixelFormat) is a more efficient (faster) destination.

Make a 32-bit RGB offscreen. See Listing 4.

Listing 3: QTNewGWorld API.

OSErr QTNewGWorld(
    GWorldPtr   *offscreenGWorld, // on return, a pointer to the GWorld
    OSType      PixelFormat; // the new GWorlds pixel format
    const Rect  *boundsRect,      // boundary and port rectangle
    CTabHandle  cTable,           // a ColorTable - NULL for default
    GDHandle    aGDevice,         // a GDevice - set to NULL
    GWorldFlags flags); // flags - set to 0 for default

Listing 4: Make a 32-bit GWorld for the decompression destination.

// Make a 32bit GWorld for the decompression destination -
// - 720 x 480 for DV. This function locks the pixel map
OSErr MakeGWorld(short inWidth, short inHeight,
                            GWorldPtr *outDestGWorld)
{
    Rect  theBounds = {0, 0};

    OSErr err = noErr;

    theBounds.right  = inWidth;
    theBounds.bottom = inHeight;
    *outDestGWorld = NULL;

    err = QTNewGWorld(outDestGWorld,     // return a pointer to
                                         // the offscreen
                     k32ARGBPixelFormat, // the new GWorlds pixel
                                         // format
                     &theBounds,         // boundry and port rect
                                         // for the offscreen PixMap
                     NULL,               // handle to a ColorTable
                     NULL,               // handle to a GDevice
                     0); // flags

    if (noErr == err)
        // call LockPixels to prevent the base address for
        // an offscreen pixel image from being moved while you
        // draw into or copy from its pixel map
        LockPixels(GetGWorldPixMap(*outDestGWorld));

    return err;
}

Back to Top

Setting up the Sequence

Perform the setup required to decompress an image sequence by calling the Image Compression Manager's DecompressSequenceBeginS function. See Listing 5. While DecompressSequenceBegin can also be used, it has been superseded by DecompressSequenceBeginS.

The DecompressSequenceBeginS call is used to initiate the sequence and specifies many of the parameters that control the sequence-decompression operation. The Image Compression Manager allocates system resources necessary for the operation, and returns a ImageSequence ID (seqID) uniquely identifying the sequence.

Start a decompression sequence. See Listing 6.

Listing 5: DecompressionSequenceBeginS API.

OSErr DecompressSequenceBeginS(
    ImageSequence          *seqID,   // returns a unique seqID
    ImageDescriptionHandle desc,     // description of
                                     // compressed data
    Ptr                    data,     // pointer to the compressed
                                     // data for preflight
    long                   dataSize, // size of the data buffer.
    CGrafPtr               port,     // the destination port
    GDHandle               gdh,      // GDevice for the destination
    const Rect             *srcRect, // portion of image to
                                     // decompress
    MatrixRecordPtr        matrix,   // transformation to apply
                                     // during decompress
    short                  mode,     // graphics transfer mode
                                     // for the operation
    RgnHandle              mask,     // mask applied during
                                     // decompression
    CodecFlags             flags,    // intermediate buffer
                                     // allocation flags
    CodecQ                 accuracy  // desired accuracy
                                     // for the operation
    DecompressorComponent  codec); // decompressor to use -
                                     // - can be special identifier

Listing 6: Signal the beginning of the process of decompressing a sequence of frames.

// Signal the beginning of the process of decompressing a
// sequence of frames Using codecHighQuality for the CodecQ
// parameter tells the decompressor to render at the highest
// image quality that can be achieved with reasonable
// performance. Using a lower CodecQ setting may be useful
// is speed is the priority.
OSErr MakeDecompressionSequence(
    ImageDescriptionHandle inImageDescription,
    GWorldPtr inDestGWorld, ImageSequence *outSeqID)
{
    Rect         theSrcBounds = {0, 0};
    Rect         theDestBounds;
    MatrixRecord rMatrix;

    *outSeqID = 0;

    if (NULL == inImageDescription) return paramErr;

    // *** IMPORTANT NOTE DV SOURCE ONLY ***
    // If your code passes in a smaller (eg. quarter-size) source
    // rectangle with the intention of drawing cropped unscaled DV to a
    // smaller (eg. quarter-size) destination, a DV-specific bug in
    // QuickTime 5 will cause this decompression request to be
    // misinterpreted, scaling the frame to fit. The correct
    // interpretationof this request is to draw the top-left corner
    // of the DV frame cropped at normal size. This bug will be
    // fixed in QuickTime 6. If your code was behaving as intended
    // because of this bug,  make sure to fix your code to use a
    // matrix in the call to DecompressSequenceBeginS, scaling the
    // frame to fit the offscreen gworld. This approach will work
    // in all versions of QuickTime.
    // *************************************

    // create a transformation matrix to scale from the source bounds
    // to the destination bounds. Using NULL for the source rectangle
    // in the call to DecompressSequenceBeginS indicates we want to
    // decompress the entire source image
    GetPortBounds(inDestGWorld, &theDestBounds);
    theSrcBounds.right  = (*inImageDescription)->width;
    theSrcBounds.bottom = (*inImageDescription)->height;

    RectMatrix(&rMatrix, &theSrcBounds, &theDestBounds);

    return DecompressSequenceBeginS(
                outSeqID,           // pointer to field to receive
                                    // unique ID for sequence
                inImageDescription, // handle to image description
                                    // structure
                NULL,               // pointer to compressed image
                                    // data (used
                                    // for preflight)
                0,                  // image data size
                inDestGWorld,       // port for the DESTINATION image
                NULL,               // grahics device handle, if
                                    // port is set, set
                                    // this to NULL
                NULL,               // source rectangle defining
                                    // the portion of the image
                                    // to decompress - NULL for
                                    // the entire source image
                &rMatrix,           // transformation matrix
                srcCopy,            // transfer mode specifier
                (RgnHandle)NULL,    // clipping region in dest.
                                    // coordinate system to use as
                                    // a mask
                0,                  // flags
                codecHighQuality,   // accuracy in decompression
                anyCodec); // compressor identifier or
                                    // special identifiers
                                    // ie. bestSpeedCodec
}

Use the GWorldPtr returned from QTNewGWorld as the destination port parameter. Using codecHighQuality for the CodecQ parameter tells the the decompressor to render at the highest image quality that can be achieved with reasonable performance. Using a lower CodecQ setting may be useful is speed is the priority.

Back to Top

Decompressing a Frame

Once the sequence has started, each frame in the sequence is queued up for decompression by calling DecompressSequenceFrameS or DecompressSequenceFrameWhen. See Listing 7. The unique ImageSequence ID returned from DecompressSequenceBeginS is passed in as the first parameter.

Listing 7: DecompressSequenceFrameWhen API.

OSErr DecompressSequenceFrameWhen(
    ImageSequence              seqID,               // unique segID
    Ptr                        data,                // pointer to
                                                    // compressed data
    long                       dataSize,            // size of the data
                                                    // buffer
    CodecFlags                 inFlags,             // control flags
    CodecFlags                 *outFlags,           // status flags
    ICMCompletionProcRecordPtr asyncCompletionProc, // async completion
                                                    // proc record
    const ICMFrameTimeRecord   *frameTime); // frame time
                                                    // information

The Image Compression Manager manages the decompression operation, calls the appropriate codec component to do the work and the frame is decompressed to the location specified in the DecompressSequenceBeginS call.

Decompress a frame. See Listing 8.

Listing 8: Simple wrapper around DecompressSequenceFrameWhen.

// A simple wrapper around DecompressSequenceFrameWhen
// Ignores in and out flags, no completion proc or
// frameTime specified - decompression operation will
// happen immediately
OSErr DecompressFrameNow(ImageSequence inSequenceID,
                        Ptr inBuffer, long inBufferSize)
{
    return DecompressSequenceFrameWhen(inSequenceID,
            inBuffer, inBufferSize, 0, NULL, NULL, NULL);
}

Listing 9: Another wrapper around DecompressSequenceFrameWhen.

// You could use a structure like this to keep track of per
// frame information
typedef struct {
    ImageSequence              seqID; // decompression
                                           // sequence ID
    Ptr                        pSrcBuffer; // pointer to
                                           // compressed data
    long                       bufSize; // compressed image
                                           // data size
    ICMCompletionProcRecordPtr
                    pCompletionProc; // pointer to a
                                           // ICMCompletionProcRecord
    Boolean                    isDestDone; // is the ICM done
                                           // with the dest?
    OSErr                      rc; // return code
} FrameRecord, *FrameRecordPtr, **FrameRecordHdl;

// Another way to wrap DecompressSequenceFrameWhen
// FrameRecordPtr assumes some type of data structure
// associated with your application which keeps track
// of per frame information
OSErr DecompressFrame(FrameRecordPtr inFrame)
{
    if (inFrame->pCompletionProc) {
        // if we set up a completion proc, store the frame
        // so it can be pulled out when we get called
        inFrame->pCompletionProc->completionRefCon = (long)inFrame;
    }

    return DecompressSequenceFrameWhen(inFrame->seqID,
                inFrame->pSrcBuffer, inFrame->bufSize,
                0, NULL, inFrame->pCompletionProc, NULL);
}

Back to Top

Asynchronous Decompression

Decompression can be performed asynchronously by specifying a completion function along with a RefCon in a ICMCompletionProcRecord. See Listing 10. In this case, make sure not to read the decompressed image until the decompressor indicates that the operation is complete by calling your completion function with the codecCompletionDest flag set. See Listing 12.

The completion function may be called multiple times and must be interrupt safe. Passing in NULL for the asyncCompletionProc specifies synchronous decompression.

Additionally, DecompressSequenceFrameWhen allows you to schedule decompression by passing in a pointer to a ICMFrameTimeRecord. See Listing 11. This structure contains parameters specifying the frame's time information, including the time at which the frame should be decompressed, its duration, and the playback rate. When implementing scheduled decompression make sure to also implement the ICMCompletionProc discussed above. This parameter can be NULL, in which case the decompression operation will happen immediately.

Listing 10: ICMCompletionProcRecord and ICMCompletionProc

// Specifies an image compression completion callback
struct ICMCompletionProcRecord {
    ICMCompletionUPP completionProc; // UPP accessing your
                                       // ICMCompletionProc
    long             completionRefCon; // refcon for use by
                                       // the callback
};
typedef struct ICMCompletionProcRecord
                    ICMCompletionProcRecord;
typedef ICMCompletionProcRecord *
                    ICMCompletionProcRecordPtr;

// Called by a compressor component upon completion
// of an asynchronous operation
typedef void (*ICMCompletionProcPtr) (OSErr result,
                                short flags, long refcon);
void MyICMCompletionProc(
    OSErr result   // result of current operation
    short flags    // flags indicating which part
                   // of the operation is complete
    long  refcon); // refcon specified in the
                   // ICMCompletionProcRecord

Listing 11: ICMFrameTimeRecord

// Contains a frame's time information for scheduled
// asynchronous decompression operations.
struct ICMFrameTimeRecord {
    wide     value; // time the frame is to be displayed
    long     scale; // units for the frame's display time
    void *   base; // the time base
    long     duration; // duration the frame is displayed
                               // must be in the same units as
                               // specified by the scale field.
                               // 0 if duration is unknown
    Fixed    rate; // the time base's effective rate
    long     recordSize; // size of this structure
    long     frameNumber; // 0 if the frame number is not known
    long     flags; // flags
    wide     virtualStartTime; // conceptual start time
    long     virtualDuration; // conceptual duration
};

Listing 12: Sample ICM Decompression Completion Procedure.

// Sample ICM decompression completion procedure
// This procedure simply checks the status of the codecCompletionDest
// flag which indicates that the Image Compression Manager is done
// with the destination buffer FrameRecordPtr assumes some type of
// data structure associated with your application which keeps
// track of per frame information
// Note: This function may be called multiple times and must be
// interrupt safe
static pascal void DecompressionDone(OSErr inResult,
                                short inFlags, long inRefCon)
{
    FrameRecordPtr theFrame = (FrameRecordPtr)inRefCon;

    if (noErr == inResult) {
        if (codecCompletionDest & inFlags) {

            // the ICM is done with the destination
            theFrame->isDestDone = true;

            ...
        }
    }

    theFrame->rc = inResult;
}

Back to Top

Accessing the Pixels

To access the pixels directly, use GetGWorldPixMap to obtain the PixMapHandle from your destination GWorld, then call GetPixBaseAddr which returns a pointer to the beginning of a pixel image.

GetPixRowBytes or QTGetPixMapHandleRowBytes can be used to retrieve the rowBytes value (the distance, in bytes, from the beginning of one row of the image data to the beginning of the next row of the image data) for a pixel map accessed by a handle, while QTGetPixMapPtrRowBytes will do the same for a pixel map accessed by a pointer. See Listing 13.

Remember to always call LockPixels to prevent the base address for an offscreen pixel image from being moved while you draw into or copy from its pixel map.

Listing 13: Accessing Pixels directly.

// You could use a structure like this for convenience
// to cast the 32bit pixel map as an array of pixels.
typedef struct {
      UInt8 alpha; // alpha component
      UInt8 red; // red component
      UInt8 green; // green component
      UInt8 blue; // blue component
} ARGBPixelRecord, *ARGBPixelPtr, **ARGBPixelHdl;

PixMapHandle hPixMap     = GetGWorldPixMap(myDestGWorld);
long         theRowBytes = QTGetPixMapHandleRowBytes(hPixMap);
Ptr          pPixels     = GetPixBaseAddr(hPixMap);

Sequence setup function. See Listing 15.

Back to Top

Ending the Sequence

After the entire sequence is decompressed, end the process by calling the CDSequenceEnd. Pass in the unique image sequence ID. See Listing 14.

Sequence end function. See Listing 16.

Listing 14: CDSequenceEnd API

OSErr CDSequenceEnd(ImageSequence seqID);

Listing 15: Sample function demonstrating setting up a decompression sequence.

// Sample function demonstrating how one might go about setting up a
// decompression sequence
// Long winded assignments are used for the purpose of this sample
// MyAppObjectPtr assumes some type of data structure associated with
// your application which keeps track of all the important bits
OSErr SetupDecompressionSequenceForDV(MyAppObjectPtr inAppObject)
{
    ImageDescriptionHandle hImageDescription = NULL;
    GWorldPtr              theGWorld = NULL;
    ImageSequence          theSequenceID = 0;
    PixMapHandle           hPixMap = NULL;
    long                   theRowBytes = 0;
    Ptr                    pPixels = NULL;

    OSErr err = noErr;

    hImageDescription = MakeImageDescriptionForDV();
    err = MemError();

    if (hImageDescription) {
        // the GWorld does not have to be the exact same size as
        // the compressed image
        err = MakeGWorld(&theGWorld,
                (*desc)->width, (*desc)->height);
        if (err) goto bail;

        err = MakeDecompressionSequence(hImageDescription,
                                theGWorld, &theSequenceID);
        if (err) goto bail;

        // get the BaseAddress and RowBytes for the pixels
        hPixMap     = GetGWorldPixMap(theGWorld);
        pPixels     = GetPixBaseAddr(hPixMap);
        theRowBytes = QTGetPixMapHandleRowBytes(hPixMap);

        // this is what we need
        inAppObject->dSeqID   = theSequenceID;
        inAppObject->pGWorld  = theGWorld;
        inAppObject->pPixels  = pPixels;
        inAppObject->rowBytes = theRowBytes;
    }

bail:
    if (hImageDescription)
        DisposeHandle((Handle)hImageDescription);

    return err;
}

Listing 16: Sample function demonstrating ending a decompression sequence.

// Sample function demonstrating how one might go about ending
// a decompression sequence MyAppObjectPtr assumes some type of
// data structure associated with your application which keeps
// track of all the important bits
OSErr EndDecompressionSequenceForDV(MyAppObjectPtr inAppObject)
{
    OSErr err = noErr;

    // end the decompression sequence
    if (0 != inAppObject->dSeqID) {
        err = CDSequenceEnd(inAppObject->dSeqID);
        inAppObject->dSeqID = 0;
    }

    // dispose of the GWorld
    if (NULL != inAppObject->pGWorld)
        DisposeGWorld(inAppObject->pGWorld);

    inAppObject->pGWorld  = NULL;
    inAppObject->pPixels  = NULL;
    inAppObject->rowBytes = 0;

    return err;
}

Back to Top

References

• QTNewGWorld

• ImageDescription Structure

• ImageDescription

• Working with Image Descriptions

• DecompressSequenceBeginS

• DecompressSequenceFrameWhen

• ICMCompletionProcRecord

• ICMCompletionProc

• ICMFrameTimeRecord

• Working with Sequences

• Image Compression Manager

• Decompressing Sequences

• Changing Sequence-Decompression Parameters

Back to Top

Document Revision History

Posted: 2002-04-11