Chapter 5. Digital Media Buffers
Prev |  | Next

## Chapter 5. Digital Media Buffers

This chapter describes the Digital Media buffers (DMbuffers) real-time visual data transport facility, which is currently supported on O2 workstations. Time-sensitive visual data is moved through system memory using DMbuffers. This method provides a unified approach to facilitating data flow between live video devices. DMbuffers provide a pipelined I/O model—the application can direct the flow of multiple images on multiple paths simultaneously. This chapter builds on video I/O concepts presented in Chapter 4, “Digital Media I/O.”

## About Digital Media Buffers

DMbuffers feature

- an operating system-generic live image data storage and transport facility

- a library-transparent interface that allows real-time data interchange with compression/decompression (dmIC) and OpenGL

- an application-centered nonblocking I/O method that provides event processing by means of a single `select()` loop

- a software-configurable memory allocation method that provides the dedicated memory and throughput resources to suit visual data transport requirements using general-purpose system memory

Because the bandwidth of digital video signals far exceeds the bandwidth of typical data storage devices and communications links, video is usually compressed when storing to disk or sending over a network. This software interface supports specialized multimedia hardware on O2 workstations which further boosts visual data processing performance.

DMbuffers and DMbufferpools provide the method for allowing your application to allocate and use general-purpose system memory for transporting visual data.

A DMbufferpool is a custom storage facility created by the application. Video I/O devices, compression devices and algorithms, and graphics devices have direct access to this storage on a compartmental basis called a DMbuffer. The application can define what this compartment represents, but in general, a DMbuffer represents an image. An image can be in the form of raw uncompressed pixel data for a video field or frame, or in the form of a picture's worth of compressed data (JPEG, MPEG, or so on).

When creating the storage facility, the application describes its data requirements in a DMparams structure. The application also queries every device that plans on using the facility about their needs, and obtains from them a DMparams list describing their requirements for allocating and apportioning memory. Using this information, the storage facility(DMbufferpool) is configured with the proper number, size, and type compartments (DMbuffers) sufficient for containing the data and its associated bookkeeping information.

Because a DMbuffer is created according to all necessary data and device requirements, and with universal content descriptors, every device that needs access to it can tell what's stored inside and can share and use the contents without making modifications.

To transfer the contents of a compartment, it is much easier and faster to transport only a pointer to the storage location rather than moving the actual contents. DMbuffers are really placeholders which contain only the pointers to the actual data; the data itself is stored elsewhere. When data is transferred, only pointers describing the data are passed, not the actual data. This means that the process doesn't have to handle or copy the data in order to transport it.

DMbufferpools contain a fixed number of DMbuffers, all the same size. It's up to you to decide how many DMbuffers to use, but, in general, three DMbuffers are sufficient for most applications: one to receive data, one to send data, and one available in case of contention between the sending and receiving processes.

DMbuffers give the application access to memory which is

- reserved at start-up and guaranteed for the life of the application

- not visited by the page daemon and therefore can't be swapped

When an image needs to be transported, a DMbuffer is allocated from the pool. Once allocated, a DMbuffer can go to a video I/O device, an image converter, or graphics I/O in any order, regardless of the order of allocation. Multiple DMbuffers with independent agendas can coexist. When the transfer is complete, a DMbuffer can be returned to the pool for reuse or retained for future use. It is not necessary to keep the same DMbuffer waiting to complete processing before accepting more input because another DMbuffer can be allocated from the pool.

Compared with the traditional VLbuffers, DMbuffers

- use the same event mechanism to deliver input and output buffer events to the application, rather than using different file descriptors for input and output

- deliver data buffers ordered with VL events on the video input path

- allow buffers to be sent to video multiple times, held by the application, or reordered

Currently, the DM buffers method is supported by:

- Image Converter Library (dmIC)

- Video Library

- Movie Library

- OpenGL

DMbuffers have the following attributes:

- Type

- Size

- MSC/UST

## DMbuffer Live Data Transport Paths

This section presents the framework for eight types of live data transport paths that can be realized using DMbuffers:

- Memory to Video

- Video to Memory

- Memory to Image Converter

- Image Converter to Memory

- Memory to Movie File

- Movie File to Memory

- Memory to OpenGL

- OpenGL to Memory

Figure 5-1 shows the eight live data transport paths that can use DMbuffers.

Figure 5-1. DMbuffer Live Data Transport Paths

Each path can be used as a stage in a more complex path. For example:

video-to-memory—>memory-to-dmIC—>dmIC-to-memory—>memory-to-movie file

are the 4 paths involved in recording compressed live video to disk.

In general, the calls outlined in this section simply move data around; they do not directly encode, decode, or otherwise process the images. This means that data is typically passed untouched along these pathways.

| Note: The outlines presented in this section are intended to provide a sketch of the calls necessary for each path; the sequence of the calls may vary within each step and the actual coding may involve more routines. See the reference pages for details about using these routines.

### Memory to Video

This section presents a basic sketch for memory to video output.

1. Open video output by calling:

   `vlOpenVideo(3dm)`, to open the video server

   `vlGetNode(3dm)`, to get the video drain node (VL_DRN, VL_VIDEO)

   `vlGetNode(3dm)`, to get the memory source node (VL_SRC, VL_MEM)

2. Create a DMbufferpool for output of video fields or frames by calling:

   `dmBufferSetPoolDefaults(3dm)`, to initialize the DMparams list

   `vlDMPoolGetParams(3dm)`, to determine the video I/O device requirements

   `dmBufferCreatePool(3dm)`, to create a pool using list modified by previous query

3. Associate the DMbufferpool with this path by calling:

   `vlDMPoolRegister(3dm)`

4. Prepare the pool to use `select()` to be notified of a free DMbuffer by calling:

   `vlGetTransferSize(3dm)`, to get the size of one field or frame in this path

   `dmBufferSetPoolSelectSize(3dm)`, to set the memory available threshold for waking from select

   `dmBufferGetPoolFD(3dm)`, to get the input file descriptor for the buffer pool

5. Get a video output path file descriptor (for notification of errors/drops) by calling:

   `vlGetFD(3dm)`

6. Start the video flow by calling:

   `vlBeginTransfer(3dm)`

7. Get notification of a free DMbuffer in the output pool by calling:

   `select(2)`

8. Fill the DMbuffer with pixel data from the desired DMbuffer path

9. Enqueue the DMbuffer for video output by calling:

   `vlDMBufferSend(3dm)`

### Video to Memory

This section presents a basic sketch for video input to memory using DMbuffers:

1. Open video input by calling:

   `vlOpenVideo(3dm)`, to open the video server

   `vlGetNode(3dm)`, to get the video source node (VL_SRC, VL_VIDEO)

   `vlGetNode(3dm)`, to get the memory drain node (VL_DRN, VL_MEM)

2. Create a DMbufferpool for input of video fields or frames by calling:

   `dmBufferSetPoolDefaults(3dm)`, to initialize the DMparams list

   `vlDMPoolGetParams(3dm)`, to determine the video I/O device requirements

   `dmBufferCreatePool(3dm)`, to create a pool with a list modified by previous query

3. Associate the DMbufferpool with this path by calling:

   `vlDMPoolRegister(3dm)`

4. Get a video input file descriptor (for use with `select(2)`) by calling:

   `vlPathGetFD(3dm)`

5. Start the video flow by calling:

   `vlSelectEvents(3dm)`, to select events, namely VLTransferCompleteMask, and VLSequenceLostMask at a minimum, and others that the process might require

   `vlBeginTransfer(3dm)`, to initiate video transfer

6. Get notification of each new field or frame by calling:

   `select(2)`, to wait for an event

7. Get a new field or frame in the form of a DMbuffer by calling:

   `vlEventRecv(3dm)`, to dequeue the next VLEvent

   `vlEventToDMBuffer(3dm)`, to convert the VLEvent into a DMbuffer

### Memory to Image Converter

1. Find the appropriate image converter by calling:

   `dmICGetNum(3dm)`, to get the number of image converters installed in the system

   `dmICGetDescription(3dm)`, to get the description of a given converter, then search them all (using the total returned) to find the index of the one that performs the desired conversion

2. Create the image converter context/instance by calling:

   `dmICCreate(3dm)`

3. Configure the image converter by calling:

   `dmSetImageDefaults(3dm)`, to initialize the DMparams list with image defaults

   `dmICSetSrcParams(3dm)`, to configure the source (input) image parameters

   `dmICSetDstParams(3dm)`, to configure the destination (output) image parameters

   `dmICSetConvParams(3dm)`, to configure the conversion algorithm (quality, etc.)

4. Create a DMbufferpool for the image converter input by calling:

   `dmBufferSetPoolDefaults(3dm)`, to initialize the DMparams list

   `dmICGetSrcPoolParams(3dm)`, to modify the list to reflect the image converter's buffering requirements (other DMbuffer paths using this pool must also be queried for their requirements before creating the buffer pool)

   `dmBufferCreatePool(3dm)`, to create a buffer pool with the required attributes

5. Prepare the pool to use `select()` to be notified of a free DMbuffer by calling:

   `dmBufferSetPoolSelectSize(3dm)`, to set the memory available threshold for waking from select

   `dmBufferGetPoolFD(3dm)`, to get the buffer pool file descriptor

6. Get notification of a free DMbuffer in the input pool by calling:

   `select(2)`, to wait for a free DMbuffer

   `dmBufferAllocate(3dm)`, to allocate a DMbuffer

7. Fill the DMbuffer with pixel data (if encoding) or bits (if decoding) by calling:

   `dmBufferMapData(3dm)`, to get a pointer to the data area of the DMbuffer

   `dmBufferSetSize(3dm)`, to set the image size, which is possibly obtained from one of: `dmImageFrameSize(3dm)`, `vlGetTransferSize(3dm)`, `mvGetTrackDataInfo(3dm)`, or `mvGetTrackDataFieldInfo(3dm)`

   `dmBufferSetUSTMSCpair(3dm)`, to set the sequence number of this image

8. Enqueue the DMbuffer for input to the converter by calling:

   `dmICSend(3dm)`

### Image Converter to Memory

1. Find the appropriate image converter by calling:

   `dmICGetNum(3dm)`, to get the number of image converters installed in the system

   `dmICGetDescription(3dm)`, to get the description of a given converter, then search them all (using the total returned) to find the index of the one that performs the desired conversion

2. Create the image converter context/instance by calling:

   `dmICCreate(3dm)`

3. Configure the image converter by calling:

   `dmSetImageDefaults(3dm)`, to initialize the DMparams list with image defaults

   `dmICSetSrcParams(3dm)`, to configure the source (input) image parameters

   `dmICSetDstParams(3dm)`, to configure the destination (output) image parameters

   `dmICSetConvParams(3dm)`, to configure the conversion algorithm (quality, etc.)

4. Create a DMbufferpool for the image converter output by calling:

   `dmBufferSetPoolDefaults(3dm)`, to initialize the DMparams list

   `dmICGetDstPoolParams(3dm)`, to modify the list to reflect the destination parameters (other DMbuffer paths using this pool must also be queried for their requirements before creating the buffer pool)

   `dmBufferCreatePool(3dm)`, to create a pool with the required attributes

   `dmICSetDstPool(3dm)`, to attach this pool to the image converter

5. Get a converter file descriptor for notification of new output from the image converter by calling:

   `dmICGetDstQueueFD(3dm)`

6. Get notification of a new DMbuffer of converted data by calling:

   `select(2)`

7. Get the new DMbuffer by calling:

   `dmICReceive(3dm)`, to dequeue the converted image from the converter

   `dmBufferGetUSTMSCpair(3dm)`, to get the sequence number of this image

   `dmBufferGetSize(3dm)`, to get the size of the image data in bytes

   `dmBufferMapData(3dm)`, to get a pointer to the image data

### Memory to Movie File

1. Open or create a movie file with an image track using one of the following methods:

   - Create a new movie file by calling:

     `mvCreateFile(3dm)`, to get a handle to the new movie file

     `mvAddTrack(3dm)`, to insert an empty image track into the movie file

   - Open an existing movie file by calling:

     `mvOpenFile(3dm)`, to get a handle to the existing movie

     `mvFindTrackByMedium(3dm)`, to find the movie's image (DM_IMAGE) track

     `mvGetTrackLength(3dm)`, to find the playing time of the current track

2. Determine the image size by calling:

   `dmBufferMapData(3dm)`, to get a pointer to the image

   `dmBufferGetSize(3dm)`, to get the size of the image in bytes

   | Note: Perform these operations twice for field-based data (once per field)

3. Save the image to the movie file using one of the following methods (pass the pointer and the size returned in the previous step to the function):

   - Write the data from a single DMbuffer as a frame by calling:

     `mvInsertTrackData(3dm)`

   - Write the data from 2 DMbuffers (one per field) by calling:

     `mvInsertTrackDataFields(3dm)`

### Movie File to Memory

1. Open a movie file for reading by calling:

   `mvOpenFile(3dm)`

   `mvFindTrackByMedium(3dm)`, to find the DM_IMAGE track

   `mvGetImageWidth(3dm)`, to get the width of the image

   `mvGetImageHeight(3dm)`, to get the height of the image

   `mvGetTrackMaxFieldSize(3dm)`, to get the size (in bytes) of the largest compressed field, which can be used for configuring DMbuffer size when creating a buffer pool

2. Get a free DMbuffer by calling:

   `dmBufferAllocate(3dm)`, to allocate a DMbuffer

   `dmBufferMapData(3dm)`, to get a pointer to the data area of the DMbuffer

3. Get the movie file index for the desired image by calling:

   `mvGetTrackDataIndexAtTime(3dm)`

4. Import the compressed image data into memory using one of the following methods:

   - If image data is stored as a frame, call:

     `mvGetTrackDataInfo(3dm)`, to get the size of the compressed data

     `mvReadTrackData(3dm)`, to copy the compressed data from the movie file directly into the DMbuffer

   - If image data is stored as fields, call:

     `mvGetTrackDataFieldInfo(3dm)`, to get the sizes of the compressed fields

     `mvReadTrackDataFields(3dm)`, to copy the 2 compressed fields directly into 2 DMbuffers

5. Set the DMbuffer size(s) by calling:

   `dmBufferSetSize(3dm)`

### Memory to OpenGL

This section presents a sketch of transporting video images to a window on the graphics display.

1. Get a pointer to the image data in a DMbuffer by calling:

   `dmBufferMapData(3dm)`

2. Display image in a graphics context (window or offscreen buffer) by calling:

   `glPixelZoom`(zoomx, -zoomy), to set a negative y zoom indicating top-to-bottom orientation (needed for video)

   `glDrawPixels(3G)`, using

   GL_YCRCB_422_SGIX, (to convert from YCrCb to RGB)

   GL_INTERLACE_SGIX (as supported—an OpenGL interlacing extension for sending two noninterlaced fields of video to an interlaced graphics display by automatically interlacing alternate lines from two video fields)

### OpenGL to Memory

The OpenGL can render to offscreen memory (pbuffer) which can be accessed directly as a DMbuffer.

1. Create an OpenGL/X pbuffer by calling:

   `glXCreateGLXPbufferSGIX()`

2. Relate the two buffers by calling:

   `glXMakeCurrent()`

   `glXAssociateDMPbufferSGIX()` with pbuffer, dmbuffer

## A Detailed Look at Recording Compressed Live Video to Disk

This section explains how to get video into memory using DMbuffers. Refer to the sample program `dmplay.dmic` in Video Capture with Compression in Chapter 4 of “Digital Media Programmer's Examples” for a demonstration of this method.

Figure 5-2 shows the video compression path.

Figure 5-2. Compression Path Using DMbuffers

The first step in video I/O is getting the video data into a DMbuffer. Once you have a buffer of video data, you can send it to dmIC to compress it, send it to graphics for display, applying effects along the way, or write it to a movie file. You can create separate threads for the video input and image conversion to ensure that the processing remains event-driven. In addition, the DMbuffers method provides the capability of performing multiple image conversion operations simultaneously within the same application with no performance penalty.

Getting video into memory requires

- a video source (input) node

- a memory drain (output) node

- a path that connects the two nodes

- an video input DMbufferpool associated with the memory drain node

This method of compressing video into memory also uses

- an input DMbufferpool on the input to the DM image converter (this is the DM buffer pool associated with the memory drain node)

- an output DMbufferpool to contain the output from the converter

The first three items are the same regardless of whether you use a DMbuffer or a VLbuffer to transport visual data, and are explained in Chapter 4.

The basic model for video input to memory using DMbuffers is:

1. Initialize the DMparams list by calling `dmBufferSetPoolDefaults(3dm).`

2. Query each device about its requirements for memory allocation:

   - Determine video I/O device requirements using `vlDMPoolGetParams(3dm)`

   - Determine image converter requirements using `dmICGetSrcPoolParams(3dm)` and `dmICGetDstPoolParams(3dm)` as described in “3. Creating Data Buffers Using the Image Conversion API” in Chapter 6.

3. Create buffer pools using the DMParams lists returned by the previous queries, by calling `dmBufferCreatePool(3dm)`.

4. Register the input buffer pool as the video input destination (drain) by calling `vlDMPoolRegister(3dm)`.

This application can be divided into the following major areas (each of which is further broken down into detailed steps):

- Video initialization, which involves

  - opening a connection to the video server (vlOpenVideo())

  - if necessary, determining which device the application will use (vlGetDevice(), vlGetDeviceList())

  - getting input information, such as the size of the video images

  | Note: Image size (in pixels) can be determined from the video timing and packing. Use `dmImageGetSize()` to determine the size of an image in bytes. The `dmplay.dmic` program first sets the capture mode to noninterlaced to get the size of each field, and then resets the capture mode to interlaced before initiating video transfer.

- Creating and setting up a video data transfer path, which involves

  - specifying nodes on the data path (vlGetNode())

  - creating a path connecting the specified nodes (vlCreatePath())

  - setting up the hardware for the path (vlSetupPaths())

- Getting and setting controls for video data transfer, which involves

  - getting input parameters (controls) for the nodes on the path (vlGetControl())

  - setting output parameters (controls) for the nodes on the path (vlSetControl())

- Creating and registering DMbufferpools to handle video data, which involves

  - initializing the DMparams list by calling `dmBufferSetPoolDefaults()`

  - querying each device about its requirements for memory allocation:

    Determine video I/O device requirements using `vlDMPoolGetParams().`

    Determine image converter requirements using `dmICGetSrcPoolParams()` and `dmICGetDstPoolParams()` as described in “3. Creating Data Buffers Using the Image Conversion API” in Chapter 6.

  - creating buffer pools using the DMParams lists modified by the previous queries, by calling `dmBufferCreatePool()`

  - registering the input buffer pool as the video input destination (drain) by calling `vlDMPoolRegister()`

- Transfer initiation, which involves:

  - setting the video transfer mode using VL_CAP_TYPE

  - specifying path-related events to be captured by calling vlSelectEvents()

  - obtaining a VL file descriptor to wait upon by calling `vlGetFD()`

  - initiating the video transfer by calling `vlBeginTransfer()`

- Main loop, which involves

  - waiting for video events using `select()`

  - dequeueing the next video event using `vlEventReceive()`

  - obtaining the video field or frame as a DMbuffer for further transport to the appropriate destination (dmIC, OpenGL, etc.) using `vlEventtoDMbuffer()`

- Cleanup, which involves

  - freeing the DMBuffers and destroying the DMbufferpools

Prev | Table of Contents | Next
Chapter 4. Digital Media I/O |  | Chapter 6. Digital Media Data Conversion