| 1 |
Caps Negotiation |
| 2 |
================ |
| 3 |
|
| 4 |
|
| 5 |
Definitions |
| 6 |
----------- |
| 7 |
|
| 8 |
- GstCaps is a structure that holds a mimetype and a set of properties. The properties are |
| 9 |
either fixed or non-fixed, they contain ranges or lists. |
| 10 |
|
| 11 |
- filter caps: caps set by the application to constrain a link to a specific GstCaps. |
| 12 |
|
| 13 |
- allowed caps: caps calculated when connecting elements, these are the types that are |
| 14 |
possible on this connection. If no types are possible, the link is refused. |
| 15 |
|
| 16 |
- pad template caps: caps put on padtemplates to describe the possible media types of |
| 17 |
the pad. |
| 18 |
|
| 19 |
- pad caps: the caps of the pad as it is currently negotiated. |
| 20 |
|
| 21 |
|
| 22 |
General overview |
| 23 |
---------------- |
| 24 |
|
| 25 |
Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure |
| 26 |
that describes the media type that will be exchanged between those pads. |
| 27 |
|
| 28 |
We define that caps negotiation only happens in the PLAYING state, when there is actual |
| 29 |
dataflow and elements thus know what kind of data they are dealing with, or when |
| 30 |
linking elements. |
| 31 |
|
| 32 |
Caps negotiation is first performed when linking elements. If the elements can agree |
| 33 |
on a media type at link time then negotiation will not have to be performed at runtime. |
| 34 |
Usually this is not possible, so the pads remain unnegotiated until at runtime. |
| 35 |
|
| 36 |
Caps negotiation is initiated by an element that wants to send data on a non-negotiated |
| 37 |
pad, thus the source pad. The core only provides a policy and convenience methods to aid |
| 38 |
the element in performing the negotiation. The core does not keep track of what pads |
| 39 |
are negotiated or not, this is a task of the pads. |
| 40 |
|
| 41 |
Caps negotiation is normally started by the source element right before it sends out |
| 42 |
data (case 1). |
| 43 |
Caps negotiation can also be redone by a sink element that wants to receive another |
| 44 |
format from a downstream element (case 2). |
| 45 |
|
| 46 |
There are two functions handling the negotiation, the link function and the do_link |
| 47 |
function. The link function is always called first and must eventually call do_link on |
| 48 |
the peer pad. The link function must figure out a compatible format for the connection |
| 49 |
and then call the do_link function on the peer pad. The do_link function can only |
| 50 |
accept or refuse the provided caps. |
| 51 |
|
| 52 |
For autopluggers it is important to know when the pad is ready to start the negotiation. |
| 53 |
It is also important to know when the negotiation failed and it must be possible to |
| 54 |
restart the negotiation with another element. This functionality will be provided |
| 55 |
with signals. |
| 56 |
|
| 57 |
Pad functions |
| 58 |
------------- |
| 59 |
|
| 60 |
! |
| 61 |
! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position); |
| 62 |
! |
| 63 |
Returns the nth possible caps that describes the media type that can flow over |
| 64 |
this pad. This function should not return static caps but caps that are |
| 65 |
dependent of the media type and the peer connections of the element. |
| 66 |
|
| 67 |
This function can be called at any time so that an autoplugger can know the |
| 68 |
exact types of the pads at any time. |
| 69 |
|
| 70 |
! |
| 71 |
! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps); |
| 72 |
! |
| 73 |
Checks that a given caps is acceptable for this pad. Returns FALSE if the |
| 74 |
pad cannot handle the caps. |
| 75 |
|
| 76 |
! |
| 77 |
! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps); |
| 78 |
! |
| 79 |
Tells the pad to start negotiation with the given filtercaps. The |
| 80 |
caps need not be fixed and serves as a filter for performing the negotiation |
| 81 |
with the peerpad. |
| 82 |
|
| 83 |
! |
| 84 |
! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps); |
| 85 |
! |
| 86 |
Configures the pad to accept data with the given caps. The caps must be fixed. |
| 87 |
|
| 88 |
! |
| 89 |
! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad); |
| 90 |
! |
| 91 |
Get the negotiated caps of a pad or NULL if the pad is not negotiated. |
| 92 |
|
| 93 |
|
| 94 |
Linking Elements |
| 95 |
---------------- |
| 96 |
|
| 97 |
When linking elements with the gst_pad_link function, the core will call |
| 98 |
the link function on the srcpad. If that pad returns GST_PAD_LINK_OK, |
| 99 |
the link is established, else the link fails. |
| 100 |
|
| 101 |
Since the link function of the pad will call the do_link function of |
| 102 |
the peerpad, the caps will be set on both pads. |
| 103 |
|
| 104 |
It is not required to decide on a caps at link time, a plugin can choose to |
| 105 |
dynamically renegotiate at runtime. |
| 106 |
|
| 107 |
When a link fails, the core emits a signal link_failed, which the application |
| 108 |
can catch to try to establish a new link with another element, for example. |
| 109 |
|
| 110 |
! |
| 111 |
! def gst_pad_link_filtered (pad1, pad2, filtercaps): |
| 112 |
! |
| 113 |
! ... get srcpad and sinkpad ... |
| 114 |
! srcpad = (pad1 == GST_PAD_SRC ? pad1 : pad2); |
| 115 |
! sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2); |
| 116 |
! |
| 117 |
! ... more checks to see if the pads are of different types and |
| 118 |
! ... that they live in the same scheduler etc... |
| 119 |
! |
| 120 |
! res = gst_pad_link (srcpad, sinkpad, filtercaps) |
| 121 |
! if (res == GST_PAD_LINK_OK) |
| 122 |
! ... perform other setup ... |
| 123 |
! else |
| 124 |
! res = signal_emit (srcpad, "link_failed") |
| 125 |
! |
| 126 |
! return res |
| 127 |
! |
| 128 |
|
| 129 |
Pad link is just a convenience function that passes a NULL filtercaps: |
| 130 |
! |
| 131 |
! def gst_pad_link (pad1, pad2): |
| 132 |
! gst_pad_link_filtered (pad1, pad2, NULL) |
| 133 |
! |
| 134 |
|
| 135 |
|
| 136 |
Dynamic renegotiation |
| 137 |
--------------------- |
| 138 |
|
| 139 |
Dynamic renegotiation happens at runtime when the element knows the exact media |
| 140 |
type it is handling. |
| 141 |
|
| 142 |
The element that wants to redecide on the data type of a connection just calls |
| 143 |
gst_pad_relink on one of it's pads. The core will call unlink and link on the |
| 144 |
pads again as if this were a new connection. Since the iterate_caps function |
| 145 |
returns other values while linking, the new link will renegotiate to a new |
| 146 |
format or the link will fail. |
| 147 |
|
| 148 |
Prototype of the relink function: |
| 149 |
! |
| 150 |
! def gst_pad_relink_filtered (pad, filtercaps): |
| 151 |
! gst_pad_unlink (pad, peerpad) |
| 152 |
! gst_pad_link_filtered (pad, peerpad, filtercaps) |
| 153 |
! |
| 154 |
|
| 155 |
Again the relink function is a convenience function for not having to pass |
| 156 |
filtercaps. |
| 157 |
! |
| 158 |
! def gst_pad_relink (pad): |
| 159 |
! gst_pad_relink_filtered (pad, NULL) |
| 160 |
! |
| 161 |
|
| 162 |
The core, however, should optimize the relink function so that it does a minimal |
| 163 |
amount of work, like not informing the scheduler about the unlink/link call in |
| 164 |
case of success. |
| 165 |
|
| 166 |
Error recovery |
| 167 |
-------------- |
| 168 |
|
| 169 |
When a pad is ready to negotiate and it has no peer pad, it fires the "need-link" |
| 170 |
signal. Autopluggers can use this signal to start plugging elements to the pad. |
| 171 |
|
| 172 |
When a link fails because of a renegotiation, the "link-failed" signal is fired |
| 173 |
so that autopluggers can try other elements. |
| 174 |
|
| 175 |
|
| 176 |
Default implementations |
| 177 |
----------------------- |
| 178 |
|
| 179 |
! |
| 180 |
! gst_pad_src_link_func (pad, peerpad, filtercaps) |
| 181 |
! { |
| 182 |
! srcpad->negotiated_caps = NULL; |
| 183 |
! gboolean possible = FALSE; |
| 184 |
! |
| 185 |
! for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++) |
| 186 |
! { |
| 187 |
! for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++) |
| 188 |
! { |
| 189 |
! test = gst_caps_intersect (srccaps, peercaps, filtercaps); |
| 190 |
! if (test == GST_CAPS_EMPTY) |
| 191 |
! continue; |
| 192 |
! |
| 193 |
! /* non empty caps, something is possible */ |
| 194 |
! possible = TRUE; |
| 195 |
! |
| 196 |
! if (!gst_caps_is_fixed (caps)) |
| 197 |
! continue; |
| 198 |
! |
| 199 |
! if (gst_pad_accepts_caps (peerpad, test)) |
| 200 |
! { |
| 201 |
! if (gst_pad_do_link (peerpad, srcpad, test)) |
| 202 |
! { |
| 203 |
! srcpad->negotiated_caps = test; |
| 204 |
! goto done; |
| 205 |
! } |
| 206 |
! else |
| 207 |
! { |
| 208 |
! /* weird, it accepted but did not want to link */ |
| 209 |
! } |
| 210 |
! } |
| 211 |
! else |
| 212 |
! { |
| 213 |
! /* caps are not accepted by peer, try next */ |
| 214 |
! } |
| 215 |
! } |
| 216 |
! } |
| 217 |
! done: |
| 218 |
! if (!possible) |
| 219 |
! return GST_PAD_LINK_FAILED; |
| 220 |
! else |
| 221 |
! return GST_PAD_LINK_OK; |
| 222 |
! } |
| 223 |
! |
| 224 |
|
| 225 |
gst_pad_iterate_caps returns caps in the following order: |
| 226 |
|
| 227 |
1) preferred caps (if any) |
| 228 |
2) negotiated caps (if any) |
| 229 |
3) profile caps (if any, filtered against caps of current media type) |
| 230 |
4) padtemplate caps (filtered against caps of current media type) |
| 231 |
|
| 232 |
1 = a caps describing the media type that would result in optimal |
| 233 |
processing of the current media type |
| 234 |
2 = a caps that it is currently handling |
| 235 |
3 = a caps describing the user configured default values |
| 236 |
4 = a caps describing all other possibilities that this pad can |
| 237 |
produce or consume given the current media type if any. |
| 238 |
|
| 239 |
|
| 240 |
generic flow 1: |
| 241 |
|
| 242 |
Linkage |
| 243 |
|
| 244 |
src sink |
| 245 |
| | |
| 246 |
| <--- link(sink, A) | |
| 247 |
check if | | |
| 248 |
A is ok | | |
| 249 |
or filter | | |
| 250 |
against | | |
| 251 |
allowed | | |
| 252 |
src caps | | |
| 253 |
| iterate_caps | |
| 254 |
if A not +-------------------------> | |
| 255 |
fixed | | |
| 256 |
| <-------------------------+ |
| 257 |
filter | | |
| 258 |
against A | | |
| 259 |
to get A' | | |
| 260 |
| can_accept(A') | |
| 261 |
if A' fix +-------------------------> | |
| 262 |
| | check if A' is ok |
| 263 |
| yes | |
| 264 |
| <-------------------------+ |
| 265 |
| | |
| 266 |
if A' not | | |
| 267 |
fixed, | | |
| 268 |
A'=null | | |
| 269 |
| do_link(src, A') | |
| 270 |
+-------------------------> | |
| 271 |
| ok | store src, A' |
| 272 |
| <-------------------------+ |
| 273 |
store | | |
| 274 |
sink, A' | | |
| 275 |
|
| 276 |
|
| 277 |
|
| 278 |
Unlink |
| 279 |
|
| 280 |
src sink |
| 281 |
| | |
| 282 |
| <--- unlink() | |
| 283 |
| | |
| 284 |
| unlink() | |
| 285 |
+-------------------------> | |
| 286 |
unref | | unref src, format |
| 287 |
sink, fmt | | |
| 288 |
|
| 289 |
|
| 290 |
Dynamic negotiation of B |
| 291 |
|
| 292 |
src sink |
| 293 |
| can_accept(B) | |
| 294 |
+-------------------------> | |
| 295 |
| | check if B is ok |
| 296 |
| yes | |
| 297 |
| <-------------------------+ |
| 298 |
| | |
| 299 |
| do_link(B) | |
| 300 |
+-------------------------> | |
| 301 |
| ok | store B |
| 302 |
| <-------------------------+ |
| 303 |
store | | |
| 304 |
B | | |
| 305 |
|
| 306 |
|
| 307 |
|
| 308 |
TODO |
| 309 |
----- |
| 310 |
|
| 311 |
Write test objects to simulate different behaviour. |
| 312 |
|
| 313 |
|
| 314 |
tests (in python): |
| 315 |
|
| 316 |
vts -> colorspace -> ximagsink |
| 317 |
|
| 318 |
relink with different colorspaces while running |
| 319 |
|
| 320 |
|
| 321 |
vts -> videoscale -> ximagesink |
| 322 |
|
| 323 |
relink to different sizes while running, scaling vts or ximagesink. |
| 324 |
|
| 325 |
sinesrc -> audiorate -> audiosink |
| 326 |
|
| 327 |
where audiosink only has one possible fixed caps. |
| 328 |
|