| 1 |
OUTDATED |
| 2 |
-------- |
| 3 |
|
| 4 |
|
| 5 |
caps negotiation |
| 6 |
================ |
| 7 |
|
| 8 |
1) purpose |
| 9 |
---------- |
| 10 |
|
| 11 |
The pads expose the media types they can handle using a mime |
| 12 |
type and a set of properties. Before the pad is created or |
| 13 |
used to pass buffers, we only know the global 'range' of media |
| 14 |
data this pad can accept. When the element has had a chance to |
| 15 |
look at the media data, only then it knows the exact values of the |
| 16 |
properties. |
| 17 |
|
| 18 |
example1: |
| 19 |
! |
| 20 |
! The mp3 decoder exposes the capabilities of its src pad |
| 21 |
! with the following caps: |
| 22 |
! |
| 23 |
! 'mpg123_src': |
| 24 |
! MIME type: 'audio/raw': |
| 25 |
! format: Integer: 16 |
| 26 |
! depth: Integer: 16 |
| 27 |
! rate: Integer range: 11025 - 48000 |
| 28 |
! channels: Integer range: 1 - 2 |
| 29 |
|
| 30 |
as you can see in example1, the padtemplate has both a range |
| 31 |
(for the audio rate) and a list (for the number of channels) |
| 32 |
for its properties. |
| 33 |
|
| 34 |
only when the mpg123 element has decoded the first mpeg audio |
| 35 |
header, it knows the exact values of the rate and channels |
| 36 |
properties. |
| 37 |
|
| 38 |
suppose that we want to connect this src pad to the sink pad |
| 39 |
of an audiosink with the properties given in example2: |
| 40 |
|
| 41 |
example2: |
| 42 |
! |
| 43 |
! 'audiosink_sink': |
| 44 |
! MIME type: 'audio/raw': |
| 45 |
! format: Integer: 16 |
| 46 |
! depth: List: |
| 47 |
! Integer: 8 |
| 48 |
! Integer: 16 |
| 49 |
! rate: Integer range: 8000 - 44000 |
| 50 |
! channels: Integer range: 1 - 2 |
| 51 |
|
| 52 |
we can see that connecting the mpg123 src pad with the |
| 53 |
audiosinks sink pad can cause a potential problem with the |
| 54 |
rate property. |
| 55 |
|
| 56 |
When the mpg123 decoder decides to output raw audio with a |
| 57 |
48000Hz samplerate, the audiosink will not be able to handle |
| 58 |
it. The conservative approach would be to disallow the connection |
| 59 |
between the two incompatible pads. This rules out any potential |
| 60 |
problems but severely limits the amount of possible connections |
| 61 |
between the elements. |
| 62 |
|
| 63 |
Another approach would be to allow the connection (and mark it |
| 64 |
as dangerous) and let the two elements figure out a suitable |
| 65 |
media type at runtime. This procedure is called caps negotiation. |
| 66 |
|
| 67 |
|
| 68 |
2) a bit of history |
| 69 |
------------------- |
| 70 |
|
| 71 |
The typing of the data that was attached to a buffer used to be |
| 72 |
done using GstMeta* (and it still is as of 11 feb 2001). With |
| 73 |
the new GstCaps and GstProps system this typing is gradually moved |
| 74 |
to the pads and to the padtemplates. This has several advantages: |
| 75 |
|
| 76 |
- the typing of the data tends to be static. The type of media |
| 77 |
doesn't change for every buffer. |
| 78 |
|
| 79 |
- Moving the typing up to the pad(templates) allows us to save |
| 80 |
them into the registry and allows us to figure out what pads |
| 81 |
are compatible. |
| 82 |
|
| 83 |
- the current metadata implementation needs header files. this may |
| 84 |
change when we also use properties for metadata. |
| 85 |
|
| 86 |
example3: |
| 87 |
! |
| 88 |
! This is the current GstMeta structure that travels with audio buffers |
| 89 |
! |
| 90 |
! struct _MetaAudioRaw { |
| 91 |
! GstMeta meta; |
| 92 |
! |
| 93 |
! /* formatting information */ |
| 94 |
! gint format; |
| 95 |
! gint channels; |
| 96 |
! gint frequency; |
| 97 |
! gint bps; |
| 98 |
! }; |
| 99 |
|
| 100 |
|
| 101 |
The question still remains what purpose the metadata will serve |
| 102 |
now that we expose the media type in the pads. Some possibilities: |
| 103 |
|
| 104 |
- interesting information, not describing the data itself but the |
| 105 |
context in which the data was generated (suggested buffer size, |
| 106 |
timestamps, etc...) |
| 107 |
|
| 108 |
- user app metadata. |
| 109 |
|
| 110 |
In this proposal we also assume that the current use of metadata using |
| 111 |
GstMeta is deprecated and that we move this information to the properties |
| 112 |
of the pads. |
| 113 |
|
| 114 |
|
| 115 |
3) the pad/padtemplates caps |
| 116 |
---------------------------- |
| 117 |
|
| 118 |
All elements have to provide a padtemplate for their pads. |
| 119 |
|
| 120 |
The padtemplates provide a range of possible media types this pad can |
| 121 |
src/sink. the main purpose for the padtemplates is to allow a |
| 122 |
rough guess at which pads are compatible before even a single buffer |
| 123 |
has been processed by the element. |
| 124 |
|
| 125 |
pads are usually created from the templates. When the pad is created |
| 126 |
it has no GstCaps* attached to it yet. The possible caps this pad |
| 127 |
can have is exposed in the padtemplate. The caps are filled in by |
| 128 |
the element when it knows the values for the caps. |
| 129 |
|
| 130 |
|
| 131 |
4) the connect function |
| 132 |
----------------------- |
| 133 |
|
| 134 |
when two pads are connected the following steps will take |
| 135 |
place (not sure, FIXME): |
| 136 |
|
| 137 |
- if both pads have caps, the caps are checked. If the caps |
| 138 |
are incompatible, the padtemplates are checked, if they |
| 139 |
are compatible, caps negotiation is performed. |
| 140 |
|
| 141 |
- if one of the pads has caps, the caps is checked against |
| 142 |
the padtemplate of the peer pad. If they are incompatible, |
| 143 |
the padtemplates are compared, if they are incompatible, |
| 144 |
caps negotiation is performed. |
| 145 |
|
| 146 |
- if none of the pads have caps, the padtemplates are checked, |
| 147 |
if they are incompatible, a warning is issued. |
| 148 |
|
| 149 |
|
| 150 |
5) when the element knows the media type it is handling |
| 151 |
------------------------------------------------------- |
| 152 |
|
| 153 |
When the element has received its first buffer it will know |
| 154 |
the media type it is handling by inspecting the buffer. |
| 155 |
|
| 156 |
before pushing the data out to its peer element(s), the element |
| 157 |
will set its src pad with the appropriate caps and properties. |
| 158 |
These caps must follow the following rules: |
| 159 |
|
| 160 |
- the caps must be compatible with the padtemplates of this |
| 161 |
pad. |
| 162 |
|
| 163 |
- the caps cannot contain ranges or lists. |
| 164 |
|
| 165 |
when the element wants to change the caps of a pad, it has to |
| 166 |
perform gst_pad_renegotiate (GstPad *pad). this will trigger |
| 167 |
the caps negotiation procedure. |
| 168 |
|
| 169 |
this will trigger the class method of the pad and calls the pads |
| 170 |
gst_pad_negotiate function: |
| 171 |
|
| 172 |
GstCaps *gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint count); |
| 173 |
|
| 174 |
This function takes a GstCaps *structure as an argument (typically the |
| 175 |
current caps of the pad) and a negotiation counter. this counter can be |
| 176 |
used to keep track of the negotiation process. |
| 177 |
|
| 178 |
The pad then creates a new caps structure with the desired caps. |
| 179 |
If the caps are accepted, it just returns the provided input caps. the |
| 180 |
_renegotiate function will set the caps of both pads whenever the |
| 181 |
input caps are the same (pointer wise) as the input caps. |
| 182 |
|
| 183 |
the caps structure is checked against the padtemplate of the peer pad, |
| 184 |
if it is incompatible the gst_pad_negotiate function is called again |
| 185 |
and the element is supposed to create another caps structure. |
| 186 |
|
| 187 |
the gst_pad_renegotiate function then calls the gst_pad_negotiate |
| 188 |
function of the peer pad with the new caps as the argument. The peer |
| 189 |
pad can adjust or create a new caps if it doesn't accept it. |
| 190 |
|
| 191 |
the caps structure keeps on bouncing between the two pads until one |
| 192 |
of the pads negotiation functions returns the caps unmodified. |
| 193 |
|
| 194 |
The element can also return a NULL pointer if it has run out of |
| 195 |
options for the caps structure. When this happens, both pads are set |
| 196 |
the NULL caps again and the pad connection is broken. |
| 197 |
|
| 198 |
The negotiation process is stopped after a fixed number of tries, |
| 199 |
when the counter has reached some limit. This limit is typically |
| 200 |
checked by the pads negotiate function. |
| 201 |
|
| 202 |
|
| 203 |
6) caps negotiation function |
| 204 |
---------------------------- |
| 205 |
|
| 206 |
the negotiate function of a pad is called whenever the pad or |
| 207 |
peer pad has performed _renegotiate. |
| 208 |
|
| 209 |
example5: |
| 210 |
! |
| 211 |
! this is the caps negotiation function implemented by an element on |
| 212 |
! one of its sink pads. |
| 213 |
! |
| 214 |
! static GstCaps* |
| 215 |
! gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint counter) |
| 216 |
! { |
| 217 |
! /* we don't accept anything else than audio/raw */ |
| 218 |
! if (strcmp (gst_caps_get_mime (caps), "audio/raw")) |
| 219 |
! return NULL; |
| 220 |
! |
| 221 |
! if (gst_caps_get_int_prop (caps, "format") != AFMT_S16_LE) |
| 222 |
! return NULL; |
| 223 |
! |
| 224 |
! /* we accept everything else */ |
| 225 |
! return caps; |
| 226 |
! } |
| 227 |
|
| 228 |
When the negotiate function returns NULL (it does not accept the |
| 229 |
specified caps of the peer pad), the negotiation process is stopped. |
| 230 |
|
| 231 |
|
| 232 |
|
| 233 |
APPENDIX A: use cases |
| 234 |
===================== |
| 235 |
|
| 236 |
1) mpg123 src!sink audiosink |
| 237 |
---------------------------- |
| 238 |
|
| 239 |
When the pads are connected the padtemplates are checked and it |
| 240 |
turns out that the pads might be incompatible (mpg123 can do |
| 241 |
48000Hz while audiosink can do 44000Hz). Nothing happens at |
| 242 |
connect time except for the user app that can mark this connection |
| 243 |
as possibly dangerous and keep some spare elements ready for when |
| 244 |
the pads turn out to be incompatible. |
| 245 |
|
| 246 |
both elements start out with no caps at all (NULL). mpg123 wants |
| 247 |
to output a buffer with specific properties. It calls |
| 248 |
gst_pad_renegotiate (mpg123->srcpad). |
| 249 |
|
| 250 |
The _renegotiate functions calls the negotiate function of the |
| 251 |
mpg123->srcpad. the negotiate function would look like this: |
| 252 |
|
| 253 |
|
| 254 |
/* |
| 255 |
* The mpg123 element cannot convert the decoded type into something |
| 256 |
* else so it has to force the caps of the src pad into the specific |
| 257 |
* type as defined by the mp3. |
| 258 |
*/ |
| 259 |
static GstCaps* |
| 260 |
gst_mpeg123_src_negotiate (GstPad *pad, GstCaps *caps, guint counter) |
| 261 |
{ |
| 262 |
GstMpg123 *mpg123; |
| 263 |
|
| 264 |
mpg123 = GST_MPG123 (gst_pad_get_parent (pad)); |
| 265 |
|
| 266 |
/* we got caps in, check them */ |
| 267 |
if (caps != NULL) { |
| 268 |
if (!strcmp (gst_caps_get_mime (caps), "audio/raw") && |
| 269 |
(gst_caps_get_int_prop (caps, "format") == AFMT_S16_LE) && |
| 270 |
(gst_caps_get_int_prop (caps, "depth") == 16) && |
| 271 |
(gst_caps_get_int_prop (caps, "rate") == mpg123->rate) && |
| 272 |
(gst_caps_get_int_prop (caps, "channels") == mpg123->channels)) { |
| 273 |
return caps; |
| 274 |
} |
| 275 |
} |
| 276 |
/* we didn't get caps, so we decide */ |
| 277 |
else if (counter != 2) { |
| 278 |
GstCaps *new; |
| 279 |
|
| 280 |
/* fill in our desired caps */ |
| 281 |
new = gst_caps_new_with_props ( |
| 282 |
"src_caps", /* name */ |
| 283 |
"audio/raw", /* mime */ |
| 284 |
gst_props_new ( |
| 285 |
"format", GST_PROPS_INT (AFMT_S16_LE), |
| 286 |
"depth", GST_PROPS_INT (16), |
| 287 |
"rate", GST_PROPS_INT (mpg123->rate), |
| 288 |
"channels", GST_PROPS_INT (mpg123->channels), |
| 289 |
NULL |
| 290 |
) |
| 291 |
); |
| 292 |
return caps; |
| 293 |
} |
| 294 |
/* too many attempts at nogotiation, bail out */ |
| 295 |
return NULL; |
| 296 |
} |
| 297 |
|
| 298 |
|
| 299 |
The audiosink pad negotiate function would look like this: |
| 300 |
|
| 301 |
/* |
| 302 |
* The audiosink has a wide range of possible parameters for |
| 303 |
* its sink pad, based on the audio card capabilities and |
| 304 |
* possibly the element configuration. |
| 305 |
* we assume the audiosink element can be both the initiator of |
| 306 |
* the negotiations and the negotiated one. |
| 307 |
*/ |
| 308 |
static GstCaps* |
| 309 |
gst_audiosink_sink_negotiate (GstPad *pad, GstCaps *caps, guint counter) |
| 310 |
{ |
| 311 |
GstAudiosink *audiosink; |
| 312 |
gboolean accepted = TRUE; |
| 313 |
|
| 314 |
audiosink = GST_AUDIOSINK (gst_pad_get_parent (pad)); |
| 315 |
|
| 316 |
/* we got caps in, we know they will match the padtemplate */ |
| 317 |
if (caps != NULL) { |
| 318 |
return caps; |
| 319 |
} |
| 320 |
/* we didn't get caps, so we decide */ |
| 321 |
else if (counter != 2) { |
| 322 |
GstCaps *new; |
| 323 |
|
| 324 |
/* fill in our desired caps */ |
| 325 |
new = gst_caps_new_with_props ( |
| 326 |
"sink_caps", /* name */ |
| 327 |
"audio/raw", /* mime */ |
| 328 |
gst_props_new ( |
| 329 |
"format", GST_PROPS_INT (audiosink->format), |
| 330 |
"depth", GST_PROPS_INT (audiosink->depth), |
| 331 |
"rate", GST_PROPS_INT (audiosink->rate), |
| 332 |
"channels", GST_PROPS_INT (audiosink->channels), |
| 333 |
NULL |
| 334 |
) |
| 335 |
); |
| 336 |
return caps; |
| 337 |
} |
| 338 |
/* too many attempts at nogotiation, bail out */ |
| 339 |
return NULL; |
| 340 |
} |
| 341 |
|
| 342 |
|
| 343 |
|
| 344 |
|
| 345 |
|
| 346 |
|
| 347 |
|
| 348 |
|
| 349 |
|