| 1 |
I'll use this doc to describe how I think media info should work from the |
| 2 |
perspective of the application developer and end user, and from that |
| 3 |
extrapolate what we need to provide that. |
| 4 |
|
| 5 |
RATIONALE |
| 6 |
--------- |
| 7 |
One of the strong points of GStreamer is that it abstracts library dependencies |
| 8 |
away. A user is free to choose whatever plug-ins he has, and a developer |
| 9 |
can code to the general API that GStreamer provides without having to deal |
| 10 |
with the underlying codecs. |
| 11 |
|
| 12 |
It is important that GStreamer also handles media info well and efficiently, |
| 13 |
since more often than not the same libraries are needed to do this. So |
| 14 |
to avoid applications depending on these libs just to do the media info, |
| 15 |
we should make sure GStreamer provides a reasonable and fast abstraction |
| 16 |
for this as well. |
| 17 |
|
| 18 |
GOALS |
| 19 |
----- |
| 20 |
- quickly read and write "tags" |
| 21 |
- quickly read stream metadata (technical properties, length, audio props, ...) |
| 22 |
- cache both kinds of data transparently |
| 23 |
- (possibly) provide bins that do this |
| 24 |
- provide a simple API to do this |
| 25 |
|
| 26 |
DEFINITION OF TERMS |
| 27 |
------------------- |
| 28 |
The user or developer using GStreamer is interested in all information that |
| 29 |
describes the stream. The library handles these two types differently |
| 30 |
however, so I will use the following terms to describe this : |
| 31 |
|
| 32 |
- metadata : |
| 33 |
every kind of information that is tied to the "concept" of the stream, |
| 34 |
and not tied to the actual encoding or representation of the stream. |
| 35 |
- it can be altered without transcoding the stream |
| 36 |
- it would stay the same for different encodings of the file |
| 37 |
- describes properties of the information encoded into the stream |
| 38 |
- examples: |
| 39 |
- artist, title, author |
| 40 |
- year, track order, album |
| 41 |
- comments |
| 42 |
|
| 43 |
- mediainfo |
| 44 |
every kind of information that is tied to the "codec" used. |
| 45 |
- cannot be altered without transcoding |
| 46 |
- is the set of parameters the stream has been encoded with |
| 47 |
- describes properties of the encoded stream itself |
| 48 |
- examples: |
| 49 |
- bitrate targets (e.g. nominal), encoding mode (e.g. joint stereo) |
| 50 |
- to this we also add "bitrate", but we query this through the pad_query |
| 51 |
interface |
| 52 |
|
| 53 |
- format |
| 54 |
every kind of information that is tied to the "raw" bitstream |
| 55 |
- cannot be altered without decoding and changing the raw bitstream |
| 56 |
- examples: |
| 57 |
- samplerate, bit depth/width, channels |
| 58 |
- length in time |
| 59 |
- video size, frames per second, colorspace used |
| 60 |
- the format is queried by getting the GstCaps of the pad that sources |
| 61 |
the buffers |
| 62 |
|
| 63 |
- length in time and tracks for the whole stream |
| 64 |
- gotten through pad queries |
| 65 |
- stored in variables in the struct |
| 66 |
|
| 67 |
- immediate info |
| 68 |
- examples: |
| 69 |
- position in time |
| 70 |
- current bitrate |
| 71 |
|
| 72 |
- tracks : |
| 73 |
a media file or stream can contain multiple consecutive streams, which |
| 74 |
we will call "tracks". GStreamer has a format for track used in querying |
| 75 |
and seeking as well. |
| 76 |
A track should be thought of as the whole of one single piece of media |
| 77 |
inside a physical stream. |
| 78 |
A track can have at most one set of tags, and has fixed "raw" properties. |
| 79 |
|
| 80 |
EXAMPLE PIPELINES |
| 81 |
----------------- |
| 82 |
reading metadata : filesrc ! id3v1 |
| 83 |
- would read metadata from file |
| 84 |
- id3v1 immediately causes filesrc to seek until it has found |
| 85 |
- the (first) metadata |
| 86 |
- that there is no metadata present |
| 87 |
- id3v1 sends out a property notification with name "metadata" and |
| 88 |
a GstCaps structure |
| 89 |
|
| 90 |
resetting and writing content metadata : |
| 91 |
id3v1 reset=true artist="Arid" ! filesink |
| 92 |
|
| 93 |
- effect: clear the current tag and reset it to only have Arid as artist |
| 94 |
- id3v1 seeks to the right location, clears the tag, and writes the new one |
| 95 |
|
| 96 |
COST |
| 97 |
---- |
| 98 |
Querying media info can be expensive. |
| 99 |
Any application querying for media info should take this into account and |
| 100 |
make sure that it doesn't block the app unnecessarily while the querying |
| 101 |
happens. |
| 102 |
|
| 103 |
The app should create an object, hand it a bunch of locations to query, |
| 104 |
and connect to the signal the app is going to send out. |
| 105 |
|
| 106 |
In most cases, querying content data should be fast since it doesn't involve |
| 107 |
decoding |
| 108 |
|
| 109 |
Technical data could be harder and thus might be better done only when needed. |
| 110 |
|
| 111 |
CACHE |
| 112 |
----- |
| 113 |
Getting media info can be an expensive operation. It makes sense to cache |
| 114 |
the dia info queried on-disk to provide rapid access to this data. |
| 115 |
It is important however that this is done transparently - the system should |
| 116 |
be able to keep working without it, or keep working when you delete this cache. |
| 117 |
|
| 118 |
The API would provide a function like |
| 119 |
gst_media_info_read_cached (media_info, location, |
| 120 |
GST_MEDIA_INFO_METADATA, |
| 121 |
GST_MEDIA_INFO_READ_CACHED); |
| 122 |
|
| 123 |
to try and get the cached metadata using the media info object. |
| 124 |
|
| 125 |
- check if the file is cached in the media info cache |
| 126 |
- if no, then read the media info and store it in the cache |
| 127 |
- if yes, then check the file against it's timestamp (or (part of) md5sum ?) |
| 128 |
- if it was changed, force a new read and store it in the cache |
| 129 |
- if it wasn't changed, just return the cached media info |
| 130 |
|
| 131 |
|
| 132 |
For optimizations, it might also make sense to do |
| 133 |
GList * gst_metadata_read_many (media_info, GList *locations, ...) |
| 134 |
|
| 135 |
which would allow the back-end to implement this more efficiently. |
| 136 |
Suppose an application loads a playlist, for example, then this playlist |
| 137 |
could be handed to this function, and a GList of metadata types could |
| 138 |
be returned. |
| 139 |
|
| 140 |
Possible implementations : |
| 141 |
- one large XML file : would end up being too heavy |
| 142 |
- one XML file per dir on system : good compromise; would still make sense |
| 143 |
to keep this in memory instead of reading and writing it all the time |
| 144 |
Also, odds are good that users mostly use files from same dir in one app |
| 145 |
(but not necessarily) |
| 146 |
|
| 147 |
Possible extra niceties : |
| 148 |
- matching of moved files, and a similar move of metadata (through user-space |
| 149 |
tool ?) |
| 150 |
|
| 151 |
!!! For speed reasons, it might make sense to somehow keep the cache in memory |
| 152 |
instead of reparsing the same cache file each time. |
| 153 |
|
| 154 |
!!! For disk space reasons, it might make sense to have a system cache. |
| 155 |
Not sure if the complexity added is worth it though. |
| 156 |
|
| 157 |
!!! For disk space reasons, we might want to add an upper limit on the size of |
| 158 |
the cache. For that we might need a timestamp on last retrieval of metadata, |
| 159 |
so that we can drop the old ones. |
| 160 |
|
| 161 |
The cache should use standard glibc. |
| 162 |
FIXME: is it worth it to use gnome-vfs for this ? |
| 163 |
|
| 164 |
STANDARDIZATION OF MEDIAINFO |
| 165 |
---------------------------- |
| 166 |
Different file formats have different "tags". It is not always possible |
| 167 |
to map metadata to tags. Some level of agreement on metadata names is also |
| 168 |
required. |
| 169 |
|
| 170 |
For media info, the names or properties should be fairly standard. |
| 171 |
We also use the same names as used for properties and capabilities in |
| 172 |
GStreamer. |
| 173 |
|
| 174 |
This means we use |
| 175 |
- encoded audio |
| 176 |
- "bitrate" (which is bits per second - use the most correct one, |
| 177 |
ie. average bitrate for VBR for example) |
| 178 |
|
| 179 |
- raw audio |
| 180 |
- "samplerate" - sampling frequency |
| 181 |
- "channels" |
| 182 |
- "bitwidth" - how wide is the audio in bits |
| 183 |
|
| 184 |
- encoded video |
| 185 |
- "bitrate" |
| 186 |
|
| 187 |
- raw video |
| 188 |
(FIXME: I don't know enough about video, are these correct) |
| 189 |
- "width" |
| 190 |
- "height" |
| 191 |
- "colordepth" |
| 192 |
- "colorspace" |
| 193 |
- "fps" |
| 194 |
- "aspectratio" |
| 195 |
|
| 196 |
We must find a way to avoid collision. A system stream can contain both |
| 197 |
audio and video (-> bitrate) or multiple audio or video streams. One way |
| 198 |
to do this might be to make a metadata set for a stream a GList of metadata |
| 199 |
for elementary streams. |
| 200 |
|
| 201 |
For metadata and tags, the standards are less clear. |
| 202 |
Some nice ones to standardize on might be |
| 203 |
- artist |
| 204 |
- title |
| 205 |
- author |
| 206 |
- year |
| 207 |
- genre (messy though) |
| 208 |
- RMS, inpoint, outpoint (calculated through some formula, used for mixing) |
| 209 |
|
| 210 |
TESTING |
| 211 |
------- |
| 212 |
It is important to write a decent testsuite for this and do speed comparisons |
| 213 |
between the library used and the GStreamer implementation. |
| 214 |
|
| 215 |
|
| 216 |
API |
| 217 |
--- |
| 218 |
struct GstMetadata |
| 219 |
{ |
| 220 |
gchar *location; |
| 221 |
GstMetadataType type; |
| 222 |
|
| 223 |
GList *streams; |
| 224 |
GHashtable *values; |
| 225 |
}; |
| 226 |
|
| 227 |
(streams would be a GList of (again) GstMetadata's. |
| 228 |
"location" would then be reused to indicate an identifier in the stream. |
| 229 |
FIXME: is that evil ?) |
| 230 |
|
| 231 |
GstMetadataType - technical, content |
| 232 |
GstMetadataReadType - cached, raw |
| 233 |
|
| 234 |
GstMetadata * |
| 235 |
gst_metadata_read (const char *location, |
| 236 |
GstMetadataType type, |
| 237 |
GstMetadataReadType read_type); |
| 238 |
GstMetadata * |
| 239 |
gst_metadata_read_props (const char *location, |
| 240 |
GList *names, |
| 241 |
GstMetadataType type, |
| 242 |
GstMetadataReadType read_type); |
| 243 |
GstMetadata * |
| 244 |
gst_metadata_read_cached (const char *location, |
| 245 |
GstMetadataType type, |
| 246 |
GstMetadataReadType read_type); |
| 247 |
|
| 248 |
GstMetadata * |
| 249 |
gst_metadata_read_props_cached (...) |
| 250 |
|
| 251 |
GList * |
| 252 |
gst_metadata_read_cached_many (GList *locations, |
| 253 |
GstMetadataType type, |
| 254 |
GstMetadataReadType read_type); |
| 255 |
|
| 256 |
GList * |
| 257 |
gst_metadata_read_props_cached_many (GList *locations, |
| 258 |
GList *names, |
| 259 |
GstMetadataType type, |
| 260 |
GstMetadataReadType read_type); |
| 261 |
|
| 262 |
GList * |
| 263 |
gst_metadata_content_write (const char *location, |
| 264 |
GstMetadata *metadata); |
| 265 |
|
| 266 |
|
| 267 |
SOME USEFUL RESOURCES |
| 268 |
--------------------- |
| 269 |
http://www.chin.gc.ca/English/Standards/metadata_multimedia.html |
| 270 |
- describes multimedia data for images |
| 271 |
distinction between content (descriptive), technical and |
| 272 |
administrative metadata |