Debian Codegraph @

docs/random/wtay/threading

raw
1 Fixing Threading
2 ----------------
3
4 1) Observations
5
6 The following observations are made when considering the current (17/11/2004)
7 problems in gstreamer.
8
9 - Bin state changes.
10 Currently the state of a bin is determined by the highest state of the
11 children, This is in particular a problem for GstThread because a thread
12 should start/stop spinning at any time depending on the state of a child.
13
14 ex 1:
15
16 +-------------------------------------+
17 | GstThread |
18 | +--------+ +---------+ +------+ |
19 | | src | | decoder | | sink | |
20 | | src-sink src-sink | |
21 | +--------+ +---------+ +------+ |
22 +-------------------------------------+
23
24 When performing the state change on the GstThread to PLAYING, one of the
25 children (at random) will go to PLAYING first, this will trigger a method
26 in GstThread that will start spinning the thread. Some elements are not yet
27 in the PLAYING state when the scheduler starts iterating elements. This
28 is not a clean way to start the data passing.
29
30 State changes also trigger negotiation and scheduling (in the other thread)
31 can do too. This creates races in negotiation.
32
33 - ERROR and EOS conditions triggering a state change
34
35 A typical problem is also that since scheduling starts while the state change
36 happens, it is possible that the elements go to EOS or ERROR before the
37 state change completes. Currently this makes the elements go to PAUSED again,
38 creating races with the state change in progress. This also gives the
39 impression to the core that the state change failed.
40
41 - no locking whatsoever
42
43 When an element does a state change, it is possible for another thread to
44 perform a conflicting state change.
45
46 - negotiation is not designed to work over multithread boundaries.
47
48 negotiation over a queue is not possible. There is no method or policy of
49 discovering a media type and then committing it. It is also not possible to
50 tie the negotiated media to the relevant buffer.
51
52 ex1:
53 it Should be possible to queue the old and the new formats in a queue.
54 The element connected to the sinkpad of the queue should be able to
55 find out that the new format will be accepted by the element connected
56 on the srcpad of the queue, even if that element is streaming the old
57 format.
58
59 +------------------------------+
60 | GstQueue |
61 | +++++++++++++++++++++++++ |
62 -sink |B|B|B|B|B|B|A|A|A|A|A|A| src-
63 | +++++++++++++++++++++++++ |
64 +------------------------------+
65 +----------+ +----------+
66 buffers in buffers in
67 new format old format
68
69 - element properties are not threadsafe
70
71 When setting an element property while streaming, the element does no
72 locking whatsoever to guarantee its internal consistency.
73
74 - No control over streaming.
75
76 When some GstThread is iterating and you want to reconnect a pad, there
77 is no way to block the pad, perform the actions and then unblock it
78 again. This leads to thread problems where a pad is negotiation at the
79 same time that it is passing data.
80
81 This is currently solved by PAUSING the pipeline or performing the actions
82 in the same threadcontext as the iterate loop.
83
84 - race conditions in synchronizing the clocks and spinning up the pipeline.
85 Currently the clock is started as soon as the pipeline is set to playing.
86 Because some time elaspes before the elements are negotiated, autoplugged
87 and streaming, the first frame/sample almost always arrives late at the
88 sinks. Hacks exist to adjust the element base time to compensate for the
89 delay but this clearly is not clean.
90
91 - race conditions when performing seeks in the pipeline. Since the elements
92 have no control over the streaming threads, they cannot block them or
93 resync them to the new seek position. It is also hard to synchronize them
94 with the clock.
95
96 - race conditions when sending tags and error messages up the pipeline
97 hierarchy. These races are either caused by glib refcounting problems and
98 by not properly locking.
99
100 - more as changes are implemented and testcases are written
101
102 2) possible solutions
103
104 - not allowing threading at all
105
106 Run the complete pipeline in a single thread. Complications to solve include
107 handling of blocking operations like source elements blocking in kernel
108 space, sink elements blocking on the clock or kernel space, etc.. In practice,
109 all operations should be made non-blocking because a blocking element can
110 cause the rest of the pipeline to block as well and cause it to miss a deadline.
111 A non-blocking model needs cooperation from the kernel (with callbacks) or
112 requires the use of a polling mechanism, both of which are either impractical
113 or too CPU intensive and in general not achievable for a general purpose
114 Multimedia framework. For this reason we will not go further with this
115 solution.
116
117 - Allow threading.
118
119 To make this work, We propose the following changes:
120
121 - Remove GstThread, it does not add anything useful in a sense that you cannot
122 arbitrarily place the thread element, it needs decoupled elements around the
123 borders.
124
125 - Simplify the state changes of bins elements. A bin or element never changes
126 state automatically on EOS and ERROR.
127
128 - Introduce the concept of the application and the streaming thread. All data
129 passing is done in the streaming thread. This also means that all operations
130 either are performed in the application thread or streaming thread and that
131 they should be protected against competing operations in other threads.
132 This would define a policy for adding appropriate locking.
133
134 - Move the creation of threads into source and loop-based elements. This will
135 make it possible for the elements in control of the threads to perform the
136 locking when needed. One particular instance is for example the state changes,
137 by creating the threads in the element, it is possible to sync the streaming
138 and the application thread (which does the state change).
139
140 - Remove negotiation from state changes. This will remove the conflict between
141 streaming and negotiating elements.
142
143 - add locks around pad operations like negotiation, streaming, linking, etc. This
144 will remove races between these conflicting operations. This will also make it
145 possible to un/block dataflow.
146
147 - add locks around bin operations like add/removing elements.
148
149 - add locks around element operations like state changes and property changes.
150
151 - add a 2-phase directed negotiation process. The source pad queries and figures
152 out what the sinkpad can take in the first phase. In the second phase it sends
153 the new format change as an event to the peer element. This event can be
154 interleaved with the buffers and can travel over queues inbetween the buffers.
155 Need to rethink this wrt bufferpools (see DShow and old bufferpool implementation)
156
157 - add a preroll phase that will be used to spin up the pipeline and align frames/samples
158 in the sinks. This phase will happen in the PAUSED state. This also means that
159 dataflow will happen in the PAUSED state. Sinks will not sink samples in the PAUSED
160 state but will complete their state change asynchronously. This will allow
161 us to have perfect synchronisation with the clock.
162
163 - a two phase seek policy. First the event travels upstream, putting all elements in
164 the seeking phase and making them synchronize to the new position. In the
165 second phase the DISCONT event signals the end of the seek and all filters can
166 continue with the new position.
167
168 - Error messages, EOS, tags and other events in the pipeline should be sent to a
169 mainloop. The app then has an in-thread mechanism for getting information about
170 the pipeline. It should also be possible to get the messages directly from the
171 elements itself, like signals. The application programmer has to know that
172 working these events come from another thread and should handle them accordingly.
173
174 - Add return values to push/pull so that errors upstream or downstream can be noted
175 by other elements so that they can disable themselves or propagate the error.
176
177
178 3) detailed explanation
179
180 a) Pipeline construction
181
182 Pipeline construction includes:
183
184 - adding/removing elements to the bin
185 - finding elements in a bin by name and interface
186 - setting the clock on a pipeline.
187 - setting properties on objects
188 - linking/unlinking pads
189
190 These operations should take the object lock to make sure it can be
191 executed from different threads.
192
193 When connecting pads to other pads from elements inside another bin,
194 we require that the bin has a ghostpad for the pad. This is needed so
195 that the bin looks like a self-contained element.
196
197 not allowed:
198 +---------------------+
199 | GstBin |
200 +---------+ | +--------+ |
201 | element | | | src | |
202 sink src------sink src- ... |
203 +---------+ | +--------+ |
204 +---------------------+
205
206 allowed:
207 +-----------------------+
208 | GstBin |
209 | +--------+ |
210 +---------+ | | src | |
211 | element | | sink src- ... |
212 sink src---sink/ +--------+ |
213 +---------+ +-----------------------+
214
215 This requirement is important when we need to sort the elements in the
216 bin to perform the state change.
217
218
219 testcases:
220
221 - create a bin, add/remove elements from it
222 - add/remove from different threads and check the bin integrity.
223
224 b) state changes
225
226 An element can be in one of the four states NULL, READY, PAUSED, PLAYING.
227
228 NULL: starting state of the element
229 READY: element is ready to start running.
230 PAUSED: element is streaming data, has opened devices etc.
231 PLAYING: element is streaming data and clock is running
232
233 Note that data starts streaming even in the PAUSED state. The only difference
234 between the PAUSED and PLAYING state is that the clock is running in the
235 PLAYING state. This mostly has an effect on the renderers which will block on
236 the first sample they receive when in PAUSED mode. The transition from
237 READY->PAUSED is called the preroll state. During that transition, media is
238 queued in the pipeline and autoplugging is done.
239
240 Elements are put in a new state using the _set_state function. This function
241 can return the following return values:
242
243 typedef enum {
244 GST_STATE_FAILURE = 0,
245 GST_STATE_PARTIAL = 1,
246 GST_STATE_SUCCESS = 2,
247 GST_STATE_ASYNC = 3
248 } GstElementStateReturn;
249
250 GST_STATE_FAILURE is returned when the element failed to go to the
251 required state. When dealing with a bin, this is returned when one
252 of the elements failed to go to the required state. The other elements
253 in the bin might have changed their states successfully. This return
254 value means that the element did _not_ change state, for bins this
255 means that not all children have changed their state.
256
257 GST_STATE_PARTIAL is returned when some elements in a bin where in the
258 locked state and therefore did not change their state. Note that the
259 state of the bin will be changed regardless of this PARTIAL return value.
260
261 GST_STATE_SUCCES is returned when all the elements successfully changed their
262 states.
263
264 GST_STATE_ASYNC is returned when an element is going to report the success
265 or failure of a state change later.
266
267 The state of a bin is not related to the state of its children but only to
268 the last state change directly performed on the bin or on a parent bin. This
269 means that changing the state of an element inside the bin does not affect
270 the state of the bin.
271
272 Setting the state on a bin that is already in the correct state will
273 perform the requested state change on the children.
274
275 Elements are not allowed to change their own state. For bins, it is allowed
276 to change the state of its children. This means that the application
277 can only know about the states of the elements it has explicitly set.
278
279 There is a difference in the way a pipeline and a bin handles the state
280 change of its children:
281
282 - a bin returns GST_STATE_ASYNC when one of its children returns an
283 ASYNC reply.
284
285 - a pipeline never returns GST_STATE_ASYNC but returns from the state
286 change function after all ASYNC elements completed the state change.
287 This is done by polling the ASYNC elements until they return their
288 final state.
289
290 The state change function must be fast an cannot block. If a blocking behaviour
291 is unavoidable, the state change function must perform an async state change.
292 Sink elements therefore always use async state changes since they need to
293 wait before the first buffer arrives at the sink.
294
295 A bin has to change the state of its children elements from the sink to the
296 source elements. This makes sure that a sink element is always ready to
297 receive data from a source element in the case of a READY->PAUSED state change.
298 In the case of a PAUSED->READY state, the sink element will be set to READY
299 first so that the source element will receive an error when it tries to push
300 data to this element so that it will shut down as well.
301
302 For loop based elements we have to be careful since they can pull a buffer
303 from the peer element before it has been put in the right state.
304 The state of a loop based element is therefore only changed after the source
305 element has been put in the new state.
306
307 c) Element state change functions
308
309 The core will call the change_state function of an element with the element
310 lock held. The element is responsible for starting any streaming tasks/threads
311 and making sure that it synchronizes them to the state change function if
312 needed.
313
314 This means that no other thread is allowed to change the state of the element
315 at that time and for bins, it is not possible to add/remove elements.
316
317 When an element is busy doing the ASYNC state change, it is possible that another
318 state change happens. The elements should be prepared for this.
319
320 An element can receive a state change for the same state it is in. This
321 is not a problem, some elements (like bins) use this to resynchronize their
322 children. Other elements should ignore this state change and return SUCCESS.
323
324 When performing a state change on an element that returns ASYNC on one of
325 the state changes, ASYNC is returned and you can only proceed to the next
326 state change when this ASYNC state change completed. Use the
327 gst_element_get_state function to know when the state change completed.
328 An example of this behaviour is setting a videosink to PLAYING, it will
329 return ASYNC in the state change from READY->PAUSED. You can only set
330 it to PLAYING when this state change completes.
331
332 Bins will perform the state change code listed in d).
333
334 For performing the state change, two variables are used: the current state
335 of the element and the pending state. When the element is not performing a
336 state change, the pending state == None. The state change variables are
337 protected by the element lock. The pending state != None as long as the
338 state change is performed or when an ASYNC state change is running.
339
340 The core provides the following function for applications and bins to
341 get the current state of an element:
342
343 bool gst_element_get_state(&state, &pending, timeout);
344
345 This function will block while the state change function is running inside
346 the element because it grabs the element lock.
347 When the element did not perform an async state change, this function returns
348 TRUE immediately with the state updated to reflect the current state of the
349 element and pending set to None.
350 When the element performed an async state change, this function will block
351 for the value of timeout and will return TRUE if the element completed the
352 async state change within that timeout, otherwise it returns FALSE, with
353 the current and pending state filled in.
354
355 The algorithm is like this:
356
357 bool gst_element_get_state(elem, &state, &pending, timeout)
358 {
359 g_mutex_lock (ELEMENT_LOCK);
360 if (elem->pending != none) {
361 if (!g_mutex_cond_wait(STATE, ELEMENT_LOCK, timeout) {
362 /* timeout triggered */
363 *state = elem->state;
364 *pending = elem->pending;
365 ret = FALSE;
366 }
367 }
368 if (elem->pending == none) {
369 *state = elem->state;
370 *pending = none;
371 ret = TRUE;
372 }
373 g_mutex_unlock (ELEMENT_LOCK);
374
375 return ret;
376 }
377
378 For plugins the following function is provided to commit the pending state,
379 the ELEMENT_LOCK should be held when calling this function:
380
381 gst_element_commit_state(element)
382 {
383 if (pending != none) {
384 state = pending;
385 pending = none;
386 }
387 g_cond_broadcast (STATE);
388 }
389
390 For bins the gst_element_get_state() works slightly different. It will run
391 the function on all of its children, as soon as one of the children returns
392 FALSE, the method returns FALSE with the state set to the current bin state
393 and the pending set to pending state.
394
395 For bins with elements that did an ASYNC state change, the _commit_state()
396 is only executed when actively calling _get_state(). The reason for this is
397 that when a child of the bin commits its state, this is not automatically
398 reported to the bin. This is not a problem since the _get_state() function
399 is the only way to get the current and pending state of the bin and is always
400 consistent.
401
402 d) bin state change algorithm
403
404 In order to perform the sink to source state change a bin must be able to sort
405 the elements. To make this easier we require that elements are connected to
406 bins using ghostpads on the bin.
407
408 The algorithm goes like this:
409
410 d = [ ] # list of delayed elements
411 p = [ ] # list of pending async elements
412 q = [ elements without srcpads ] # sinks
413 while q not empty do
414 e = dequeue q
415 s = [ all elements connected to e on the sinkpads ]
416 q = q append s
417 if e is entry point
418 d = d append e
419 else
420 r = state change e
421 if r is ASYNC
422 p = p append e
423 done
424 while d not empty do
425 e = dequeue d
426 r = state change e
427 if r is ASYNC
428 p = p append e
429 done
430 # a bin would return ASYNC here if p is not empty
431
432 # this last part is only performed by a pipeline
433 while p not empty do
434 e = peek p
435 if state completed e
436 dequeue e from p
437 done
438
439 The algorithm first tries to find the sink elements, ie. ones without
440 sinkpads. Then it changes the state of each sink elements and queues
441 the elements connected to the sinkpads.
442
443 The entry points (loopbased and getbased elements) are delayed as we
444 first need to change the state of the other elements before we can activate
445 the entry points in the pipeline.
446
447 The pipeline will poll the async children before returning.
448
449 e) The GstTask infrastructure
450
451 A new component: GstTask is added to the core. A task is created by
452 an instance of the abstract GstScheduler class.
453
454 Each schedulable element (when added to a pipeline) is handed a
455 reference to a GstScheduler. It can use this object to create
456 a GstTask, which is basically a managed wrapper around a threading
457 library like GThread. It should be possible to write a GstScheduler
458 instance that uses other means of scheduling, like one that does not
459 use threads but implements task switching based on mutex locking.
460
461 When source and loopbased elements want to create the streaming thread
462 they create an instance of a GstTask, which they pass a pointer to
463 a loop-function. This function will be called as soon as the element
464 performs GstTask.start(). The element can stop and uses mutexes to
465 pause the GstTask from, for example, the state change function or the
466 event functions.
467
468 The GstTasks implement the streaming threads.
469
470 f) the preroll phase
471
472 Element start the streaming threads in the READY->PAUSED state. Since
473 the elements that start the threads are put in the PAUSED state last,
474 after their connected elements, they will be able to deliver data to
475 their peers without problems.
476
477 Sink elements like audio and videosinks will return an async state change
478 reply and will only commit the state change after receiving the first
479 buffer. This will implement the preroll phase.
480
481 The following pseudo code shows an algorithm for committing the state
482 change in the streaming method.
483
484 GST_OBJECT_LOCK (element);
485 /* if we are going to PAUSED, we can commit the state change */
486 if (GST_STATE_TRANSITION (element) == GST_STATE_READY_TO_PAUSED) {
487 gst_element_commit_state (element);
488 }
489 /* if we are paused we need to wait for playing to continue */
490 if (GST_STATE (element) == GST_STATE_PAUSED) {
491
492 /* here we wait for the next state change */
493 do {
494 g_cond_wait (element->state_cond, GST_OBJECT_GET_LOCK (element));
495 } while (GST_STATE (element) == GST_STATE_PAUSED);
496
497 /* check if we got playing */
498 if (GST_STATE (element) != GST_STATE_PLAYING) {
499 /* not playing, we can't accept the buffer */
500 GST_OBJECT_UNLOCK (element);
501 gst_buffer_unref (buf);
502 return GST_FLOW_WRONG_STATE;
503 }
504 }
505 GST_OBJECT_UNLOCK (element);
506
507
508
509 g) return values for push/pull
510
511 To recover from pipeline errors in a more elegant manner than just
512 shutting down the pipeline, we need more finegrained error messages
513 in the data transport. The plugins should be able to know what goes
514 wrong when interacting with their outside environment. This means
515 that gst_pad_push/gst_pad_pull and gst_event_send should return a
516 result code.
517
518 Possible return values include:
519
520 - GST_OK
521 - GST_ERROR
522 - GST_NOT_CONNECTED
523 - GST_NOT_NEGOTIATED
524 - GST_WRONG_STATE
525 - GST_UNEXPECTED
526 - GST_NOT_SUPPORTED
527
528 GST_OK
529 Data transport was successful
530
531 GST_ERROR
532 An error occurred during transport, such as a fatal decoding error,
533 the pad should not be used again.
534
535 GST_NOT_CONNECTED
536 The pad was not connected
537
538 GST_NOT_NEGOTIATED
539 The peer does not know what datatype is going over the pipeline.
540
541 GST_WRONG_STATE
542 The peer pad is not in the correct state.
543
544 GST_UNEXPECTED
545 The peer pad did not expect the data because it was flushing or
546 received an eos.
547
548 GST_NOT_SUPPORTED
549 The operation is not supported.
550
551 The signatures of the functions will become:
552
553 GstFlowReturn gst_pad_push (GstPad *pad, GstBuffer *buffer);
554 GstFlowReturn gst_pad_pull (GstPad *pad, GstBuffer **buffer);
555
556 GstResult gst_pad_push_event (GstPad *pad, GstEvent *event);
557
558 - push_event will send the event to the connected pad.
559
560 For sending events from the application:
561
562 GstResult gst_pad_send_event (GstPad *pad, GstEvent *event);
563
564 h) Negotiation
565
566 Implement a simple two phase negotiation. First the source queries the
567 sink if it accepts a certain format, then it sends the new format
568 as an event. Sink pads can also trigger a state change by requesting
569 a renegotiation.
570
571 i) Mainloop integration/GstBus
572
573 All error, warning and EOS messages from the plugins are sent to an event
574 queue. The pipeline reads the messages from the queue and will either
575 handle them or forward them to the main event queue that is read by the
576 application.
577
578 Specific pipelines can be written that deal with negotiation messages and
579 errors in the pipeline intelligently. The basic pipeline will stop the
580 pipeline when an error occurs.
581
582 Whenever an element posts a message on the event queue, a signal is also
583 fired that can be caught by the application. When dealing with those
584 signals the application has to be aware that they come from the streaming
585 threads and need to make sure they use proper locking to protect their
586 own data structures.
587
588 The messages will be implemented using a GstBus object that allows
589 plugins to post messages and allows the application to read messages either
590 synchronous or asynchronous. It is also possible to integrate the bus in
591 the mainloop.
592
593 The messages will derive from GstData to make them a lightweight refcounted
594 object. Need to figure out how we can extend this method to encapsulate
595 generic signals in messages too.
596
597 This decouples the streaming thread from the application thread and should
598 avoid race conditions and pipeline stalling due to application interaction.
599
600 It is still possible to receive the messages in the streaming thread context
601 if an application wants to. When doing this, special care has to be taken
602 when performing state changes.
603
604 j) EOS
605
606 When an element goes to EOS, it sends the EOS event to the peer plugin
607 and stops sending data on that pad. The peer element that received an EOS
608 event on a pad can refuse any buffers on that pad.
609
610 All elements without source pads must post the EOS message on the message
611 queue. When the pipeline receives an EOS event from all sinks, it will
612 post the EOS message on the application message queue so that the application
613 knows the pipeline is in EOS. Elements without any connected sourcepads
614 should also post the EOS message. This makes sure that all "dead-ends"
615 signalled the EOS.
616
617 No state change happens when elements go to EOS but the elements with the
618 GstTask will stop their tasks and so stop producing data.
619
620 An application can issue a seek operation which makes all tasks running
621 again so that they can start streaming from the new location.
622
623
624
625 A) threads and lowlatency
626
627 People often think it is a sin to use threads in low latency applications. This is true
628 when using the data has to pass thread boundaries but false when it doesn't. Since
629 source and loop based elements create a thread, it is possible to construct a pipeline
630 where data passing has to cross thread boundaries, consider this case:
631
632 +-----------------------------------+
633 | +--------+ +--------+ |
634 | |element1| |element2| |
635 | .. -sink src-sink src- .. |
636 | +--------+ +--------+ |
637 +-----------------------------------+
638
639 The two elements are loop base and thus create a thread to drive the pipeline. At the
640 border between the two elements there is a mutex to pass the data between the two
641 threads. When using these kinds of element in a pipeline, low-latency will not be
642 possible. For low-latency apps, don't use these constructs!
643
644 Note that in a typical pipeline with one get-based element and two chain-based
645 elements (decoder/sink) there is only one thread, no data is crossing thread
646 boundaries and thus this pipeline can be low-latency. Also note that while this
647 pipeline is streaming no interaction or locking is done between it and the main
648 application.
649
650 +-------------------------------------+
651 | +--------+ +---------+ +------+ |
652 | | src | | decoder | | sink | |
653 | | src-sink src-sink | |
654 | +--------+ +---------+ +------+ |
655 +-------------------------------------+
656
657
658 B) howto make non-threaded pipelines
659
660 For low latency it is required to not have datapassing cross any thread
661 borders. Here are some pointers for making sure this requirement is met:
662
663 - never connect a loop or chain based element to a loop based element, this
664 will create a new thread for the sink loop element.
665
666 - do not use queues or any other decoupled element, as they implicitly
667 create a thread boundary.
668
669 - At least one thread will be created for any source element (either in the
670 connected loop-based element or in the source itself) unless the source
671 elements are connected to the same loop based element.
672
673 - when designing sinks, make them non-blocking, use the async clock callbacks
674 to schedule media rendering in the same thread (if any) as the clock. Sinks that
675 provide the clock can be made blocking.
676