docs: add draft for arbitrary buffer metadata idea

1 files changed, 382 insertions, 0 deletions

diff --git a/docs/design/draft-buffer2.txt b/docs/design/draft-buffer2.txt
new file mode 100644
index 0000000000..d55fce0f40
--- /dev/null
+++ b/docs/design/draft-buffer2.txt
@@ -0,0 +1,382 @@
+GstBuffer^2
+-----------
+
+This draft document describes a possible design for arbitrary per-buffer
+metadata.
+
+The proposed changes in this document are not ABI/API compatible with the 0.10
+version of GStreamer and should thus only be considered for upcomming unstable
+versions.
+
+Buffer metadata typically includes properties that give more information about
+the buffer contents. These properties are usually not negotiated and are thus
+not inside the caps.
+
+Some examples of metadata:
+
+ - timestamp, duration
+ - offset, offset_end
+ - interlacing information
+ - video alignment, cropping, panning information
+ - extra container information such as granulepos, ...
+ - extra global buffer properties
+
+
+Requirements
+------------
+
+ - It must be fast
+    * allocation, free, low fragmentation
+    * access to the metadata fields, preferably not much slower than directly
+      accessing a C structure field
+ - It must be extensible. Elements should be able to add new arbitrary metadata
+   without requiring much effort. Also new metadata fields should not break API
+   or ABI.
+ - It plays nice with subbuffers. When a subbuffer is created, the various
+   buffer metadata should be copied/updated correctly.
+ - We should be able to pass metadata in pad_alloc() and get_range() functions
+   to specify extra allocation parameters.
+ - We should be able to attach statically allocated metadata to a buffer. This
+   is for metadata that does not change much.
+
+
+GstMiniObject
+-------------
+
+We make GstMiniObject a simple refcounted C structure and also a GLib boxed
+type. The following fields will be in the structure:
+
+struct _GstMiniObject {
+  GType type;
+
+  /*< public >*/ /* with COW */
+  /* refcounting */
+  gint       refcount;
+  guint      flags;
+
+  GstMiniObjectCopyFunction copy;
+  GstMiniObjectFreeFunction free;
+}
+
+We will use the regular GSlice allocator or custom object pooling for allocating
+instances of the mini object.
+
+We use the well known refcounting mechanisms to manage the lifetime of the
+objects.
+
+
+GstEvent, GstCaps, GstQuery, GstMessage
+---------------------------------------
+
+Have the new GstMiniObject be the first field in these objects. They will probably
+also replace the copy and free functions with their own implementations.
+
+Allocation of the objects will use the regular gst_*_new() functions that will
+allocate and initialize a parent GstMiniObject of the required size and setting up
+the custom functions.
+
+
+GstBuffer
+---------
+
+A GstMiniObject will be the parent instance of the GstBuffer object, which is a
+regular C structure.
+
+struct _GstBuffer {
+  GstMiniObject          mini_object;
+
+  gsize                  free_size;
+
+  GstCaps               *caps;
+  GstBuffer             *parent;
+
+  gpointer _gst_padding[10];
+};
+
+The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
+as a first class feature of a GstBuffer.
+
+Allocation of the GstBuffer structure will result in the allocation of a memory region
+of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
+region will initially be used. The remaining bytes will be part of the free metadata
+region of the buffer. The size of the free region is kept in the free_size field.
+
+Buffers point to a GstCaps structure that contains the caps of the buffer data.
+
+
+GstBufferMeta
+-------------
+
+A GstBufferMeta is a structure as follows:
+
+  struct _GstBufferMeta {
+    GstBufferMetaInfo *info;  /* tag and info for the meta item */
+    GstBufferMeta     *next;  /* pointer to the next item */
+  }
+
+The purpose of the this structure is to serve as a common header for all metadata
+information that we can attach to a buffer. Specific metadata, such as timing metadata,
+will have this structure as the first field. For example:
+
+  struct _GstBufferMetaTiming {
+    GstBufferMeta  meta;        /* common meta header */
+ 
+    GstClockTime   dts;         /* decoding timestamp */
+    GstClockTime   pts;         /* presentation timestamp */
+    GstClockTime   duration;    /* duration of the data */
+    GstClockTime   clock_rate;  /* clock rate for the above values */
+  };
+
+Or another example for the buffer memory region
+
+  struct _GstBufferMetaMemory {
+    GstBufferMeta      meta;
+   
+    /* pointer to data and its size */
+    guint8            *data;
+    guint              size;
+    guint8            *malloc_data;
+    GFreeFunc          data_free;
+    gpointer           data_user;
+  };
+
+GstBufferMetaInfo will point to more information about the metadata and looks like this:
+
+  struct _GstBufferMetaInfo {
+    GQuark                     tag;       /* tag name */
+    gsize                      size;      /* size of the structure */
+
+    GstMetaInitFunction        init_func;
+    GstMetaFreeFunction        free_func;
+    GstMetaSubFunction         sub_func;
+    GstMetaSerializeFunction   serialize_func
+    GstMetaDeserializeFunction deserialize_func
+  }
+
+Tag will contain a GQuark of the metadata name. We will be able to refer to specific
+metadata by name or by its (cached) GQuark.  A repository of registered MetaInfo
+will be maintained by the core. We will register some common metadata structures
+in core and some media specific info for audio/video/text in -base. Plugins can
+register additional custom metadata.
+
+Along with the metadata description we will have functions to initialize/free (and/or refcount)
+a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer
+function that can be used to modify the metadata when a subbuffer is taken.
+
+We also add serialize and deserialize function for the metadata in case we need special
+logic for reading and writing the metadata. This is needed for GDP payloading of the
+metadata.
+
+The purpose of the separate MetaInfo is to not have to carry the free/init functions in
+each buffer instance but to define them globally. We still want quick access to the info
+so we need to make the buffer metadata point to the info.
+
+Technically we could also specify the field and types in the MetaInfo and
+provide a generic API to retrieve the metadata fields without the need for a
+header file. We will not do this yet.
+ 
+The complete buffer with metadata would then look as follows:
+
+                         +-------------------------------------+
+GstMiniObject            |     GType (GstBuffer)               |
+                         |     refcount, flags, copy/free      |
+                         +-------------------------------------+
+GstBuffer                |     caps, parent, subfunc           |
+                         +.....................................+
+                      +- |     info                           ------> GstBufferMetaInfo
+GstBufferMetaTiming   |  |     next                           ---+
+                      |  |                                     | |
+                      |  |     dts                             | |
+                      |  |     pts                             | |
+                      |  |     duration                        | |
+                      +- |     clock_rate                      | |
+                         + . . . . . . . . . . . . . . . . . . + |
+                      +- |     info                           <--+ -> GstBufferMetaInfo
+GstBufferMetaMemory   |  |     next                           ---+
+                      |  |                                     | |
+                      |  |     data                            | |
+                      |  |     size                            | |
+                      |  |     mallocdata                      | |
+                      |  |     data_free                       | |
+                      +- |     data_user                       | |
+                         + . . . . . . . . . . . . . . . . . . + .
+                         .                                       .
+
+API examples
+------------
+
+Buffers are created using the normal gst_buffer_new functions. The standard fields
+are initialized as usual. A memory area that is bigger than the structure size
+is allocated for the buffer metadata. The remaining free area is stored in the
+free_size field.
+
+  gst_buffer_new ();
+
+After creating a buffer, the application can set caps. and add other metadata
+information. 
+
+In order to modify metadata, a reference to the MetaInfo should be obtained.
+This can be done like this:
+
+  GstBufferMetaInfo *info;
+
+  info = gst_buffer_meta_get_info (GQuark tag);
+
+Usually the info will be obtained only once in order to avoid lock contention on
+the global pool of meta info. The core will also provide convenience functions
+for the core metainfo.
+
+Once a reference to the info has been obtained, the associated metadata can be
+added or modified on a buffer.
+
+For example, to modify the timing info on a buffer, one could use the following
+sequence:
+
+  GstBufferMetaInfo *info;
+  GstBufferMetaTiming *timing;
+
+  info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK);
+  
+  timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */
+  timing->timestamp = 0;
+  timing->duration = 20 * GST_MSECOND;
+
+The _get_meta() function returns a pointer to the metadata structure associated
+with the GST_META_TIMING_QUARK info.
+
+For the core meta info, we will provide convenience code that uses the cached
+GstBufferMetaInfo, making the above code a little more simple.
+
+  GstBufferMetaTiming *timing;
+
+  timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
+  timing->timestamp = 0;
+  timing->duration = 20 * GST_MSECOND;
+ 
+Note that for each of the metadata that we will add to buffers, we need a struct
+definition and a registered MetaInfo. 
+
+
+We will also provide an API to iterate the different metainfo structures. A
+possible simple API would look like this:
+
+ GstBufferMeta *current = NULL;
+
+ /* passing NULL gives the first entry */ 
+ current = gst_buffer_meta_get_next (buffer, current);
+
+ /* passing a GstBufferMeta returns the next */
+ current = gst_buffer_meta_get_next (buffer, current);
+
+
+
+Memory management
+-----------------
+
+* allocation
+
+  We will initially allocate a reasonable sized GstBuffer structure (say 512
+  bytes) and we will set the free_size to the maximum amount of metadata we can
+  store.
+
+  Since the complete buffer structure, including a large area for metadata, is
+  allocated in one go, we can reduce the number of memory allocations while still
+  providing dynamic metadata.
+
+  When adding metadata, we need to call the init function of the associated
+  metadata info structure. Since adding the metadata requires the caller to pass
+  a handle to the info, this operation does not require table lookups.
+
+  Per-metadata memory initialisation is needed because not all metadata is
+  initialized in the same way. We need to, for example, set the timestamps to
+  NONE in the MetaTiming structures.
+
+  The init/free functions can also be used to implement refcounting for a metadata
+  structure. This can be useful is a structure is shared between buffers.
+
+  When the free_size of the GstBuffer is exhausted, we will allocate new memory
+  for each newly added BufferMeta and use the next pointers to point to this. It
+  is expected that this does not occur often and we might be able to optimize
+  this transparently in the future.
+
+* free
+
+  When a GstBuffer is freed, we potentially might have to call a custom free
+  function on the metadata info. In the case of the Memory metadata, we need to
+  call the associated free function to free the memory.
+  
+  When freeing a GstBuffer, the custom buffer free function will iterate all of
+  the metadata in the buffer and call the associated free functions in the
+  MetaInfo associated with the entries. Usually, this function will be NULL.
+
+
+Subbufers
+---------
+
+Subbuffers are a first class feature of the GstBuffer. 
+
+Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
+parent buffer. It will then iterate all of the metadata entries for the parent
+buffer and call the associated sub_func in the MetaInfo.
+
+This allows each metadata structure to implement the actions needed to update
+the metadata of the subbuffer. 
+
+A pointer to the old and new memory location of the metadata is passed to the
+sub_func. The default implementation will simply copy the metadata. Custom
+implementations can adjust the values. For example, when making a subbuffer, the
+timing metadata needs to be reset to NONE when the start offset is different.
+
+
+Serialization
+-------------
+
+When buffer should be sent over the wire or be serialized in GDP, we need a way
+to perform custom serialization and deserialization on the metadata.
+
+For this we add the serialize and deserialize functions to the metadata info.
+Possible use cases are to make sure we write out the fields with a specific size
+and endianness.
+
+
+Other use cases
+---------------
+
+Making the GstBufferMetaMemory (for making the buffer point to the associated
+memory region) as metadata on a GstBuffer, as opposed to making it an integral
+part of GstBuffer, allows for some more interesting ways to transfer data.
+
+We could for example make a new GstBufferMetaIOVec metadata structure like this:
+
+  struct _GstBufferMetaIOVec {
+    GstBufferMeta  meta;
+   
+    /* pointer to data and its size */
+    GFreeFunc       data_free;
+    gpointer        data_user;
+    guint           len;
+    struct iovec   *iov;
+  };
+
+This would allow us to transfer data in a scatter/gather array. Since the fields
+in the buffer metadata are now explicit, elements that don't support this kind
+of metadata can gracefully degrade.
+
+Another use case for not having the Memory metadata in the buffers would be for
+_pad_alloc() and get_range(). We can pass a GstBuffer with the requested
+metadata fields to those functions and have the _get_range() or pad_alloc()
+implementations add (or use, in the case of a file reader) the memory metadata. 
+
+
+Relationship with GstCaps
+-------------------------
+
+The difference between GstCaps, used in negotiation, and the metadata is not
+clearly defined. 
+
+We would like to think of the GstCaps containing the information needed to
+functionally negotiate the format between two elements. The Metadata should then
+only contain variables that can change between each buffer.
+
+
+