Debian Codegraph @

doc/intro/tutorials/work_with_spyder.rst

raw
1 .. _tutorial_spyder:
2
3 :octicon:`book;1em;sd-text-info` DataLab and Spyder: a perfect match
4 ====================================================================
5
6 .. meta::
7 :description: Tutorial showing how to use Spyder to work with DataLab, the open-source data analysis and visualization platform
8 :keywords: DataLab, Spyder, tutorial, data analysis, data visualization, data processing, data science, scientific computing, scientific visualization
9
10 This tutorial demonstrates how to use `Spyder`_ to work with DataLab through a
11 practical example, using sample algorithms and data that represent a hypothetical
12 research or technical workflow. The goal is to illustrate how to use DataLab to
13 test your algorithms with data and debug them when necessary.
14
15 The example is straightforward, yet it demonstrates the essential concepts of
16 working with both DataLab *and* `Spyder`_.
17
18 .. image:: ../../images/tutorials/work_with_spyder/spyder_and_datalab.png
19 :class: dark-light
20 :alt: DataLab and Spyder: a perfect match
21
22
23 .. note::
24
25 DataLab and `Spyder`_ are **complementary** tools. While `Spyder`_ is a powerful
26 development environment with interactive scientific computing capabilities,
27 DataLab is a versatile data analysis platform that can perform a wide range
28 of tasks, from simple data visualization to complex data analysis and
29 processing. In other words, `Spyder`_ is a **development** tool, while
30 DataLab is a **data analysis** tool. You can use `Spyder`_ to develop
31 algorithms and then use DataLab to analyze data with those algorithms.
32
33 Basic concepts
34 --------------
35
36 In the context of your research or technical work, we assume that you are developing
37 software to process data (signals or images). This software may be a stand-alone
38 application, a library for use in other applications, or even a simple script to
39 run from the command line. In any case, you will need to follow a development
40 process that typically includes the following steps:
41
42 0. Prototype the algorithm in a development environment such as `Spyder`_.
43 1. Develop the algorithm in a development environment such as `Spyder`_.
44 2. Test the algorithm with data.
45 3. Debug the algorithm if necessary.
46 4. Repeat steps 2 and 3 until the algorithm works as expected.
47 5. Use the algorithm in your application.
48
49 .. note::
50
51 DataLab can help you with step 0 because it provides all the processing
52 primitives you need to prototype your algorithm: you can load data, visualize
53 it, and perform basic processing operations. We won't cover this step in the
54 following sections since the DataLab documentation already provides extensive
55 information about it.
56
57 In this tutorial, we will see how to use DataLab to perform steps 2 and 3. We assume
58 that you have already prototyped (preferably in DataLab!) and developed your algorithm
59 in `Spyder`_. Now, you want to test it with data, but without leaving `Spyder`_
60 because you may need to modify your algorithm and re-test it. Besides, your workflow
61 is already set up in `Spyder`_ and you don't want to change it.
62
63 .. note::
64
65 This tutorial assumes that you have already installed DataLab and started it.
66 If you haven't done so yet, please refer to the :ref:`installation` section
67 of the documentation.
68
69 Additionally, we assume that you have already installed `Spyder`_ and started it.
70 If you haven't done so yet, please refer to the `Spyder`_ documentation.
71 **Note that you don't need to install DataLab in the same environment as Spyder:**
72 that's the whole point of DataLab—it is a stand-alone application that can be
73 used from any environment. For this tutorial, you only need to install the
74 Sigima package (``pip install sigima``) in the same environment as `Spyder`_.
75
76 Testing your algorithm with DataLab
77 -----------------------------------
78
79 Let's assume that you have developed algorithms in the ``my_work`` module of your
80 project. You have already prototyped them in DataLab and developed them in `Spyder`_
81 by writing functions that take data as input and return processed data as output.
82 Now, you want to test these algorithms with data.
83
84 To test these algorithms, you have written two functions in the ``my_work`` module:
85
86 * ``test_my_1d_algorithm``: this function returns 1D data that will allow you
87 to validate your first algorithm, which processes 1D data.
88
89 * ``test_my_2d_algorithm``: this function returns 2D data that will allow you
90 to validate your second algorithm, which processes 2D data.
91
92 You can now use DataLab to visualize the data returned by these functions directly
93 from `Spyder`_:
94
95 - First, start both DataLab and `Spyder`_.
96 - Remember that DataLab is a stand-alone application that can be used from any
97 environment, so you don't need to install it in the same environment as `Spyder`_
98 because the connection between these two applications is established through a
99 communication protocol.
100
101 Here is how to do it:
102
103 .. literalinclude:: my_work_test_with_datalab.py
104 :language: python
105 :start-after: # ---- Begin of the example code ----
106
107 If we execute the first two cells, we will see the following output in the `Spyder`_
108 console:
109
110 .. code-block:: python
111
112 Python 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)]
113 Type "copyright", "credits" or "license" for more information.
114
115 IPython 8.15.0 -- An enhanced Interactive Python.
116
117 In [1]: runcell('Connecting to DataLab current session', 'my_work_test_with_datalab.py')
118 Connecting to DataLab XML-RPC server...OK (port: 54577)
119
120 In [2]: runcell('Visualizing 1D data from my work', 'my_work_test_with_datalab.py')
121 Out[2]: True
122
123 .. figure:: ../../images/tutorials/work_with_spyder/01.png
124
125 In this screenshot, we can see the result of evaluating the first two cells: the
126 first cell connects to DataLab, and the second cell visualizes the 1D data returned
127 by the ``test_my_1d_algorithm`` function.
128
129 .. figure:: ../../images/tutorials/work_with_spyder/02.png
130
131 In this screenshot, we can see the result of evaluating the third cell: the
132 ``test_my_2d_algorithm`` function returns a 2D array, which we can visualize
133 directly in DataLab.
134
135 .. _debug_spyder:
136
137 Debugging your algorithm with DataLab
138 -------------------------------------
139
140 Now that you have tested your algorithms with data, you may want to debug them
141 if necessary. To do so, you can combine the `Spyder`_ debugging capabilities with
142 DataLab.
143
144 Here is the code of the sample algorithm that we want to debug, in which we have
145 introduced an optional ``debug_with_datalab`` parameter that—if set to ``True``—
146 will create a proxy object allowing step-by-step visualization of the data in DataLab:
147
148 .. literalinclude:: my_work.py
149 :language: python
150 :pyobject: generate_2d_data
151
152 The corresponding ``test_my_2d_algorithm`` function also has an optional
153 ``debug_with_datalab`` parameter that is simply passed to the ``generate_2d_data``
154 function.
155
156 Now, we can use `Spyder`_ to debug the ``test_my_2d_algorithm`` function:
157
158 .. literalinclude:: my_work_debug_with_datalab.py
159 :language: python
160 :start-after: # ---- Begin of the example code ----
161
162 In this simple example, the algorithm iterates 10 times and generates a 1D array
163 at each iteration. Each 1D array is then stacked into a 2D array that is returned
164 by the ``generate_2d_data`` function. With the ``debug_with_datalab`` parameter
165 set to ``True``, we can visualize each 1D array in DataLab: that way, we can
166 verify that the algorithm is working as expected.
167
168 .. figure:: ../../images/tutorials/work_with_spyder/03.png
169
170 In this screenshot, we can see the result of evaluating the first cell: the
171 ``test_my_2d_algorithm`` function is called with the ``debug_with_datalab``
172 parameter set to ``True``: 10 1D arrays are generated and visualized in DataLab.
173
174 .. note::
175
176 If we had executed the script using the `Spyder`_ debugger and set a breakpoint in
177 the ``generate_2d_data`` function, we would have seen the generated 1D arrays
178 in DataLab at each iteration. Since DataLab runs in a separate process, we would
179 have been able to interact with the data in DataLab while the algorithm is paused
180 in `Spyder`_.
181
182 .. _Spyder: https://www.spyder-ide.org/