From dd670cb5b5cef8b3ec24db7fbc4eaf6ff775b565 Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Fri, 10 Jul 2009 18:46:39 +0100 Subject: docs: add draft for arbitrary buffer metadata idea --- docs/design/draft-buffer2.txt | 382 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 382 insertions(+) create mode 100644 docs/design/draft-buffer2.txt (limited to 'docs') diff --git a/docs/design/draft-buffer2.txt b/docs/design/draft-buffer2.txt new file mode 100644 index 0000000000..d55fce0f40 --- /dev/null +++ b/docs/design/draft-buffer2.txt @@ -0,0 +1,382 @@ +GstBuffer^2 +----------- + +This draft document describes a possible design for arbitrary per-buffer +metadata. + +The proposed changes in this document are not ABI/API compatible with the 0.10 +version of GStreamer and should thus only be considered for upcomming unstable +versions. + +Buffer metadata typically includes properties that give more information about +the buffer contents. These properties are usually not negotiated and are thus +not inside the caps. + +Some examples of metadata: + + - timestamp, duration + - offset, offset_end + - interlacing information + - video alignment, cropping, panning information + - extra container information such as granulepos, ... + - extra global buffer properties + + +Requirements +------------ + + - It must be fast + * allocation, free, low fragmentation + * access to the metadata fields, preferably not much slower than directly + accessing a C structure field + - It must be extensible. Elements should be able to add new arbitrary metadata + without requiring much effort. Also new metadata fields should not break API + or ABI. + - It plays nice with subbuffers. When a subbuffer is created, the various + buffer metadata should be copied/updated correctly. + - We should be able to pass metadata in pad_alloc() and get_range() functions + to specify extra allocation parameters. + - We should be able to attach statically allocated metadata to a buffer. This + is for metadata that does not change much. + + +GstMiniObject +------------- + +We make GstMiniObject a simple refcounted C structure and also a GLib boxed +type. The following fields will be in the structure: + +struct _GstMiniObject { + GType type; + + /*< public >*/ /* with COW */ + /* refcounting */ + gint refcount; + guint flags; + + GstMiniObjectCopyFunction copy; + GstMiniObjectFreeFunction free; +} + +We will use the regular GSlice allocator or custom object pooling for allocating +instances of the mini object. + +We use the well known refcounting mechanisms to manage the lifetime of the +objects. + + +GstEvent, GstCaps, GstQuery, GstMessage +--------------------------------------- + +Have the new GstMiniObject be the first field in these objects. They will probably +also replace the copy and free functions with their own implementations. + +Allocation of the objects will use the regular gst_*_new() functions that will +allocate and initialize a parent GstMiniObject of the required size and setting up +the custom functions. + + +GstBuffer +--------- + +A GstMiniObject will be the parent instance of the GstBuffer object, which is a +regular C structure. + +struct _GstBuffer { + GstMiniObject mini_object; + + gsize free_size; + + GstCaps *caps; + GstBuffer *parent; + + gpointer _gst_padding[10]; +}; + +The Buffer object will contain a pointer to the parent buffer to allow for subbuffers +as a first class feature of a GstBuffer. + +Allocation of the GstBuffer structure will result in the allocation of a memory region +of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this +region will initially be used. The remaining bytes will be part of the free metadata +region of the buffer. The size of the free region is kept in the free_size field. + +Buffers point to a GstCaps structure that contains the caps of the buffer data. + + +GstBufferMeta +------------- + +A GstBufferMeta is a structure as follows: + + struct _GstBufferMeta { + GstBufferMetaInfo *info; /* tag and info for the meta item */ + GstBufferMeta *next; /* pointer to the next item */ + } + +The purpose of the this structure is to serve as a common header for all metadata +information that we can attach to a buffer. Specific metadata, such as timing metadata, +will have this structure as the first field. For example: + + struct _GstBufferMetaTiming { + GstBufferMeta meta; /* common meta header */ + + GstClockTime dts; /* decoding timestamp */ + GstClockTime pts; /* presentation timestamp */ + GstClockTime duration; /* duration of the data */ + GstClockTime clock_rate; /* clock rate for the above values */ + }; + +Or another example for the buffer memory region + + struct _GstBufferMetaMemory { + GstBufferMeta meta; + + /* pointer to data and its size */ + guint8 *data; + guint size; + guint8 *malloc_data; + GFreeFunc data_free; + gpointer data_user; + }; + +GstBufferMetaInfo will point to more information about the metadata and looks like this: + + struct _GstBufferMetaInfo { + GQuark tag; /* tag name */ + gsize size; /* size of the structure */ + + GstMetaInitFunction init_func; + GstMetaFreeFunction free_func; + GstMetaSubFunction sub_func; + GstMetaSerializeFunction serialize_func + GstMetaDeserializeFunction deserialize_func + } + +Tag will contain a GQuark of the metadata name. We will be able to refer to specific +metadata by name or by its (cached) GQuark. A repository of registered MetaInfo +will be maintained by the core. We will register some common metadata structures +in core and some media specific info for audio/video/text in -base. Plugins can +register additional custom metadata. + +Along with the metadata description we will have functions to initialize/free (and/or refcount) +a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer +function that can be used to modify the metadata when a subbuffer is taken. + +We also add serialize and deserialize function for the metadata in case we need special +logic for reading and writing the metadata. This is needed for GDP payloading of the +metadata. + +The purpose of the separate MetaInfo is to not have to carry the free/init functions in +each buffer instance but to define them globally. We still want quick access to the info +so we need to make the buffer metadata point to the info. + +Technically we could also specify the field and types in the MetaInfo and +provide a generic API to retrieve the metadata fields without the need for a +header file. We will not do this yet. + +The complete buffer with metadata would then look as follows: + + +-------------------------------------+ +GstMiniObject | GType (GstBuffer) | + | refcount, flags, copy/free | + +-------------------------------------+ +GstBuffer | caps, parent, subfunc | + +.....................................+ + +- | info ------> GstBufferMetaInfo +GstBufferMetaTiming | | next ---+ + | | | | + | | dts | | + | | pts | | + | | duration | | + +- | clock_rate | | + + . . . . . . . . . . . . . . . . . . + | + +- | info <--+ -> GstBufferMetaInfo +GstBufferMetaMemory | | next ---+ + | | | | + | | data | | + | | size | | + | | mallocdata | | + | | data_free | | + +- | data_user | | + + . . . . . . . . . . . . . . . . . . + . + . . + +API examples +------------ + +Buffers are created using the normal gst_buffer_new functions. The standard fields +are initialized as usual. A memory area that is bigger than the structure size +is allocated for the buffer metadata. The remaining free area is stored in the +free_size field. + + gst_buffer_new (); + +After creating a buffer, the application can set caps. and add other metadata +information. + +In order to modify metadata, a reference to the MetaInfo should be obtained. +This can be done like this: + + GstBufferMetaInfo *info; + + info = gst_buffer_meta_get_info (GQuark tag); + +Usually the info will be obtained only once in order to avoid lock contention on +the global pool of meta info. The core will also provide convenience functions +for the core metainfo. + +Once a reference to the info has been obtained, the associated metadata can be +added or modified on a buffer. + +For example, to modify the timing info on a buffer, one could use the following +sequence: + + GstBufferMetaInfo *info; + GstBufferMetaTiming *timing; + + info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK); + + timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */ + timing->timestamp = 0; + timing->duration = 20 * GST_MSECOND; + +The _get_meta() function returns a pointer to the metadata structure associated +with the GST_META_TIMING_QUARK info. + +For the core meta info, we will provide convenience code that uses the cached +GstBufferMetaInfo, making the above code a little more simple. + + GstBufferMetaTiming *timing; + + timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */ + timing->timestamp = 0; + timing->duration = 20 * GST_MSECOND; + +Note that for each of the metadata that we will add to buffers, we need a struct +definition and a registered MetaInfo. + + +We will also provide an API to iterate the different metainfo structures. A +possible simple API would look like this: + + GstBufferMeta *current = NULL; + + /* passing NULL gives the first entry */ + current = gst_buffer_meta_get_next (buffer, current); + + /* passing a GstBufferMeta returns the next */ + current = gst_buffer_meta_get_next (buffer, current); + + + +Memory management +----------------- + +* allocation + + We will initially allocate a reasonable sized GstBuffer structure (say 512 + bytes) and we will set the free_size to the maximum amount of metadata we can + store. + + Since the complete buffer structure, including a large area for metadata, is + allocated in one go, we can reduce the number of memory allocations while still + providing dynamic metadata. + + When adding metadata, we need to call the init function of the associated + metadata info structure. Since adding the metadata requires the caller to pass + a handle to the info, this operation does not require table lookups. + + Per-metadata memory initialisation is needed because not all metadata is + initialized in the same way. We need to, for example, set the timestamps to + NONE in the MetaTiming structures. + + The init/free functions can also be used to implement refcounting for a metadata + structure. This can be useful is a structure is shared between buffers. + + When the free_size of the GstBuffer is exhausted, we will allocate new memory + for each newly added BufferMeta and use the next pointers to point to this. It + is expected that this does not occur often and we might be able to optimize + this transparently in the future. + +* free + + When a GstBuffer is freed, we potentially might have to call a custom free + function on the metadata info. In the case of the Memory metadata, we need to + call the associated free function to free the memory. + + When freeing a GstBuffer, the custom buffer free function will iterate all of + the metadata in the buffer and call the associated free functions in the + MetaInfo associated with the entries. Usually, this function will be NULL. + + +Subbufers +--------- + +Subbuffers are a first class feature of the GstBuffer. + +Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the +parent buffer. It will then iterate all of the metadata entries for the parent +buffer and call the associated sub_func in the MetaInfo. + +This allows each metadata structure to implement the actions needed to update +the metadata of the subbuffer. + +A pointer to the old and new memory location of the metadata is passed to the +sub_func. The default implementation will simply copy the metadata. Custom +implementations can adjust the values. For example, when making a subbuffer, the +timing metadata needs to be reset to NONE when the start offset is different. + + +Serialization +------------- + +When buffer should be sent over the wire or be serialized in GDP, we need a way +to perform custom serialization and deserialization on the metadata. + +For this we add the serialize and deserialize functions to the metadata info. +Possible use cases are to make sure we write out the fields with a specific size +and endianness. + + +Other use cases +--------------- + +Making the GstBufferMetaMemory (for making the buffer point to the associated +memory region) as metadata on a GstBuffer, as opposed to making it an integral +part of GstBuffer, allows for some more interesting ways to transfer data. + +We could for example make a new GstBufferMetaIOVec metadata structure like this: + + struct _GstBufferMetaIOVec { + GstBufferMeta meta; + + /* pointer to data and its size */ + GFreeFunc data_free; + gpointer data_user; + guint len; + struct iovec *iov; + }; + +This would allow us to transfer data in a scatter/gather array. Since the fields +in the buffer metadata are now explicit, elements that don't support this kind +of metadata can gracefully degrade. + +Another use case for not having the Memory metadata in the buffers would be for +_pad_alloc() and get_range(). We can pass a GstBuffer with the requested +metadata fields to those functions and have the _get_range() or pad_alloc() +implementations add (or use, in the case of a file reader) the memory metadata. + + +Relationship with GstCaps +------------------------- + +The difference between GstCaps, used in negotiation, and the metadata is not +clearly defined. + +We would like to think of the GstCaps containing the information needed to +functionally negotiate the format between two elements. The Metadata should then +only contain variables that can change between each buffer. + + + -- cgit v1.2.3