| 1 |
.. _tutorial_spyder: |
| 2 |
|
| 3 |
:octicon:`book;1em;sd-text-info` DataLab and Spyder: a perfect match |
| 4 |
==================================================================== |
| 5 |
|
| 6 |
.. meta:: |
| 7 |
:description: Tutorial showing how to use Spyder to work with DataLab, the open-source data analysis and visualization platform |
| 8 |
:keywords: DataLab, Spyder, tutorial, data analysis, data visualization, data processing, data science, scientific computing, scientific visualization |
| 9 |
|
| 10 |
This tutorial demonstrates how to use `Spyder`_ to work with DataLab through a |
| 11 |
practical example, using sample algorithms and data that represent a hypothetical |
| 12 |
research or technical workflow. The goal is to illustrate how to use DataLab to |
| 13 |
test your algorithms with data and debug them when necessary. |
| 14 |
|
| 15 |
The example is straightforward, yet it demonstrates the essential concepts of |
| 16 |
working with both DataLab *and* `Spyder`_. |
| 17 |
|
| 18 |
.. image:: ../../images/tutorials/work_with_spyder/spyder_and_datalab.png |
| 19 |
:class: dark-light |
| 20 |
:alt: DataLab and Spyder: a perfect match |
| 21 |
|
| 22 |
|
| 23 |
.. note:: |
| 24 |
|
| 25 |
DataLab and `Spyder`_ are **complementary** tools. While `Spyder`_ is a powerful |
| 26 |
development environment with interactive scientific computing capabilities, |
| 27 |
DataLab is a versatile data analysis platform that can perform a wide range |
| 28 |
of tasks, from simple data visualization to complex data analysis and |
| 29 |
processing. In other words, `Spyder`_ is a **development** tool, while |
| 30 |
DataLab is a **data analysis** tool. You can use `Spyder`_ to develop |
| 31 |
algorithms and then use DataLab to analyze data with those algorithms. |
| 32 |
|
| 33 |
Basic concepts |
| 34 |
-------------- |
| 35 |
|
| 36 |
In the context of your research or technical work, we assume that you are developing |
| 37 |
software to process data (signals or images). This software may be a stand-alone |
| 38 |
application, a library for use in other applications, or even a simple script to |
| 39 |
run from the command line. In any case, you will need to follow a development |
| 40 |
process that typically includes the following steps: |
| 41 |
|
| 42 |
0. Prototype the algorithm in a development environment such as `Spyder`_. |
| 43 |
1. Develop the algorithm in a development environment such as `Spyder`_. |
| 44 |
2. Test the algorithm with data. |
| 45 |
3. Debug the algorithm if necessary. |
| 46 |
4. Repeat steps 2 and 3 until the algorithm works as expected. |
| 47 |
5. Use the algorithm in your application. |
| 48 |
|
| 49 |
.. note:: |
| 50 |
|
| 51 |
DataLab can help you with step 0 because it provides all the processing |
| 52 |
primitives you need to prototype your algorithm: you can load data, visualize |
| 53 |
it, and perform basic processing operations. We won't cover this step in the |
| 54 |
following sections since the DataLab documentation already provides extensive |
| 55 |
information about it. |
| 56 |
|
| 57 |
In this tutorial, we will see how to use DataLab to perform steps 2 and 3. We assume |
| 58 |
that you have already prototyped (preferably in DataLab!) and developed your algorithm |
| 59 |
in `Spyder`_. Now, you want to test it with data, but without leaving `Spyder`_ |
| 60 |
because you may need to modify your algorithm and re-test it. Besides, your workflow |
| 61 |
is already set up in `Spyder`_ and you don't want to change it. |
| 62 |
|
| 63 |
.. note:: |
| 64 |
|
| 65 |
This tutorial assumes that you have already installed DataLab and started it. |
| 66 |
If you haven't done so yet, please refer to the :ref:`installation` section |
| 67 |
of the documentation. |
| 68 |
|
| 69 |
Additionally, we assume that you have already installed `Spyder`_ and started it. |
| 70 |
If you haven't done so yet, please refer to the `Spyder`_ documentation. |
| 71 |
**Note that you don't need to install DataLab in the same environment as Spyder:** |
| 72 |
that's the whole point of DataLab—it is a stand-alone application that can be |
| 73 |
used from any environment. For this tutorial, you only need to install the |
| 74 |
Sigima package (``pip install sigima``) in the same environment as `Spyder`_. |
| 75 |
|
| 76 |
Testing your algorithm with DataLab |
| 77 |
----------------------------------- |
| 78 |
|
| 79 |
Let's assume that you have developed algorithms in the ``my_work`` module of your |
| 80 |
project. You have already prototyped them in DataLab and developed them in `Spyder`_ |
| 81 |
by writing functions that take data as input and return processed data as output. |
| 82 |
Now, you want to test these algorithms with data. |
| 83 |
|
| 84 |
To test these algorithms, you have written two functions in the ``my_work`` module: |
| 85 |
|
| 86 |
* ``test_my_1d_algorithm``: this function returns 1D data that will allow you |
| 87 |
to validate your first algorithm, which processes 1D data. |
| 88 |
|
| 89 |
* ``test_my_2d_algorithm``: this function returns 2D data that will allow you |
| 90 |
to validate your second algorithm, which processes 2D data. |
| 91 |
|
| 92 |
You can now use DataLab to visualize the data returned by these functions directly |
| 93 |
from `Spyder`_: |
| 94 |
|
| 95 |
- First, start both DataLab and `Spyder`_. |
| 96 |
- Remember that DataLab is a stand-alone application that can be used from any |
| 97 |
environment, so you don't need to install it in the same environment as `Spyder`_ |
| 98 |
because the connection between these two applications is established through a |
| 99 |
communication protocol. |
| 100 |
|
| 101 |
Here is how to do it: |
| 102 |
|
| 103 |
.. literalinclude:: my_work_test_with_datalab.py |
| 104 |
:language: python |
| 105 |
:start-after: # ---- Begin of the example code ---- |
| 106 |
|
| 107 |
If we execute the first two cells, we will see the following output in the `Spyder`_ |
| 108 |
console: |
| 109 |
|
| 110 |
.. code-block:: python |
| 111 |
|
| 112 |
Python 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)] |
| 113 |
Type "copyright", "credits" or "license" for more information. |
| 114 |
|
| 115 |
IPython 8.15.0 -- An enhanced Interactive Python. |
| 116 |
|
| 117 |
In [1]: runcell('Connecting to DataLab current session', 'my_work_test_with_datalab.py') |
| 118 |
Connecting to DataLab XML-RPC server...OK (port: 54577) |
| 119 |
|
| 120 |
In [2]: runcell('Visualizing 1D data from my work', 'my_work_test_with_datalab.py') |
| 121 |
Out[2]: True |
| 122 |
|
| 123 |
.. figure:: ../../images/tutorials/work_with_spyder/01.png |
| 124 |
|
| 125 |
In this screenshot, we can see the result of evaluating the first two cells: the |
| 126 |
first cell connects to DataLab, and the second cell visualizes the 1D data returned |
| 127 |
by the ``test_my_1d_algorithm`` function. |
| 128 |
|
| 129 |
.. figure:: ../../images/tutorials/work_with_spyder/02.png |
| 130 |
|
| 131 |
In this screenshot, we can see the result of evaluating the third cell: the |
| 132 |
``test_my_2d_algorithm`` function returns a 2D array, which we can visualize |
| 133 |
directly in DataLab. |
| 134 |
|
| 135 |
.. _debug_spyder: |
| 136 |
|
| 137 |
Debugging your algorithm with DataLab |
| 138 |
------------------------------------- |
| 139 |
|
| 140 |
Now that you have tested your algorithms with data, you may want to debug them |
| 141 |
if necessary. To do so, you can combine the `Spyder`_ debugging capabilities with |
| 142 |
DataLab. |
| 143 |
|
| 144 |
Here is the code of the sample algorithm that we want to debug, in which we have |
| 145 |
introduced an optional ``debug_with_datalab`` parameter that—if set to ``True``— |
| 146 |
will create a proxy object allowing step-by-step visualization of the data in DataLab: |
| 147 |
|
| 148 |
.. literalinclude:: my_work.py |
| 149 |
:language: python |
| 150 |
:pyobject: generate_2d_data |
| 151 |
|
| 152 |
The corresponding ``test_my_2d_algorithm`` function also has an optional |
| 153 |
``debug_with_datalab`` parameter that is simply passed to the ``generate_2d_data`` |
| 154 |
function. |
| 155 |
|
| 156 |
Now, we can use `Spyder`_ to debug the ``test_my_2d_algorithm`` function: |
| 157 |
|
| 158 |
.. literalinclude:: my_work_debug_with_datalab.py |
| 159 |
:language: python |
| 160 |
:start-after: # ---- Begin of the example code ---- |
| 161 |
|
| 162 |
In this simple example, the algorithm iterates 10 times and generates a 1D array |
| 163 |
at each iteration. Each 1D array is then stacked into a 2D array that is returned |
| 164 |
by the ``generate_2d_data`` function. With the ``debug_with_datalab`` parameter |
| 165 |
set to ``True``, we can visualize each 1D array in DataLab: that way, we can |
| 166 |
verify that the algorithm is working as expected. |
| 167 |
|
| 168 |
.. figure:: ../../images/tutorials/work_with_spyder/03.png |
| 169 |
|
| 170 |
In this screenshot, we can see the result of evaluating the first cell: the |
| 171 |
``test_my_2d_algorithm`` function is called with the ``debug_with_datalab`` |
| 172 |
parameter set to ``True``: 10 1D arrays are generated and visualized in DataLab. |
| 173 |
|
| 174 |
.. note:: |
| 175 |
|
| 176 |
If we had executed the script using the `Spyder`_ debugger and set a breakpoint in |
| 177 |
the ``generate_2d_data`` function, we would have seen the generated 1D arrays |
| 178 |
in DataLab at each iteration. Since DataLab runs in a separate process, we would |
| 179 |
have been able to interact with the data in DataLab while the algorithm is paused |
| 180 |
in `Spyder`_. |
| 181 |
|
| 182 |
.. _Spyder: https://www.spyder-ide.org/ |