Debian Codegraph @

debian/TODO

raw
1 -*- mode: org -*-
2 * general
3 ** DONE libtbx_env
4 A discussion about this is available on github
5 https://github.com/cctbx/cctbx_project/issues/460
6
7 - pytest load libtbx.env_config qui unpickle the build environment.
8 We need to patch the code and install the pickled under /etc.
9
10 /usr/lib/python3/dist-packages/libtbx/env_config.py:2627: in
11 unpickle build_path = os.environ["LIBTBX_BUILD"]
12 /usr/lib/python3.8/os.py:675: in __getitem__ raise KeyError(key)
13 from None E KeyError: 'LIBTBX_BUILD'
14
15 for now the path could be: `/var/lib/libtbx/libtbx_env`
16
17 - [X] create a python script which fix the libtbx_env for final
18 installed version. this script will be run during the build
19 process and installed in place of the original one. We keep the
20 original one for the build tests.
21
22 ** DONE take care of the python modules.
23 - [X] patch to remove pathlib2
24 ** TODO bin wrappers.
25 the generated wrappers looks like this.
26
27 #+begin_src sh
28 LIBTBX_PYEXE="/usr/bin/$LIBTBX_PYEXE_BASENAME"
29 export LIBTBX_PYEXE
30 if [ -n "$LIBTBX__VALGRIND_FLAG__" ]; then
31 exec $LIBTBX_VALGRIND "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py" "$@"
32 elif [ $# -eq 0 ]; then
33 exec "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py"
34 else
35 exec "$LIBTBX_PYEXE" "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py" "$@"
36 fi
37 #+end_src
38
39 the first problem is the path of the script, which is relative to
40 $LIBTBX_BUILD.
41
42 This variable is extracted from
43
44 #+begin_src sh
45 # ----------------------------------------------------------------------------
46 # The shellrealpath function resolves an absolute physical path of its
47 # first argument and stores it in a global shell variable RESULT.
48 # The function returns nonzero for unreadable or invalid symlinks
49 # and resets the RESULT to an empty string.
50
51 shellrealpath() {
52 local ORGDIR="$PWD"
53 local TARGET="$1"
54 RESULT=""
55 # This test fails for a symlink loop. We can do without resolution
56 # of symlinks that point to existing unreadable files.
57 [ -r "$TARGET" ] || return $?
58 # Check if the readlink command exists.
59 type readlink >/dev/null || return $?
60 while true; do
61 cd "$(dirname "$TARGET")"
62 TARGET="$(basename "$TARGET")"
63 if [ -L "$TARGET" ]; then
64 TARGET="$(readlink "$TARGET")"
65 continue
66 fi
67 RESULT="$(pwd -P)/$TARGET"
68 break
69 done
70 cd "$ORGDIR"
71 }
72 # ----------------------------------------------------------------------------
73 LIBTBX_BUILD="$(shellrealpath "$0" && cd "$(dirname "$RESULT")/.." && pwd)"
74
75 #+end_src
76
77 the value of this variable is "/usr" our case.
78
79 Another problem is that once installer on the system, the source path is wrong.
80
81 "$LIBTBX_BUILD/../../../../modules/cctbx_project/libtbx/command_line/show_env.py"
82
83 we install the modules here
84
85 /usr/lib/python3/dist-packages/libtbx/command_line/show_env.py
86
87 an easy fix (but fragile) would be to replace the values in all the
88 bin scripts via sed, but this does not guarantees that these wrong
89 path ar not use somewhere else in the code.
90
91 the interest of this solution is that we do not touch the upstream
92 generated code and the tests can be used out of the box.
93
94 beware there is two cases. the first on is for the cctbx_project
95 modules and the other one is for the dials modules (without the
96 cctbx_project)
97
98 it is necessary also to remove the LD_LIBRARY_PATH part, PATH and
99 PYTHONPATH.
100 #+begin_src sh
101 :~/Debian/dials/dials$ dials.find_spots
102 Traceback (most recent call last):
103 File "/usr/lib/python3/dist-packages//dials/command_line/find_spots.py", line 9, in <module>
104 from dials.array_family import flex
105 File "/usr/lib/python3/dist-packages/dials/array_family/flex.py", line 137, in <module>
106 from dials.array_family.flex_ext import ( # noqa: F401; lgtm
107 File "/usr/lib/python3/dist-packages/dials/array_family/flex_ext.py", line 18, in <module>
108 import cctbx.miller
109 File "/usr/lib/python3/dist-packages/cctbx/miller/__init__.py", line 13, in <module>
110 from cctbx import maptbx
111 File "/usr/lib/python3/dist-packages/cctbx/maptbx/__init__.py", line 17, in <module>
112 import libtbx.load_env
113 File "/usr/lib/python3/dist-packages/libtbx/load_env.py", line 5, in <module>
114 libtbx.env = libtbx.env_config.unpickle()
115 File "/usr/lib/python3/dist-packages/libtbx/env_config.py", line 2630, in unpickle
116 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
117 FileNotFoundError: [Errno 2] No such file or directory: '/usr/libtbx_env'
118 #+end_src
119
120 This is because the LIBTBX_BUILD variable is exported in all the
121 wrappers. When LIBTBX_BUILD is exported it is used to find the
122 libtbx_env pickled environment. This is why it does not work. In
123 that case build_path=/usr
124
125 It seems that we must use libtbx.refresh in order to regenerate all
126 the scripts.
127
128 But it does not work because my script dos not install the non
129 =.py= files. So I missed all the .sh scripts.
130
131 ** TODO script use tcsh instead of sh
132 - [X] create setup.py for cctbx and dials
133 - [ ] modify them to install also the data.
134 ** TODO hardening
135 - [X] all the libraries and python extensions are aware of the
136 hardening flags.
137 - [ ] upstream the patche used to fix a bunch of missing flags.
138 ** TODO build the documentation
139 ** TODO autocompletion
140 autocompletion are generate under /build/dials/autocomplete
141 ** TODO dials.find_spots does not work
142 https://github.com/dials/dials/issues/1262 I need to recompile
143 bitschuffle with hdf5-serial. It seems that there is a strange
144 interaction between OpenMP and dials/cctbx. I think that hdf5
145 should not be forgotten in the equation. I learn that cctbx
146 provide an omptbx which is a stub, which should not be used because
147 it breaks OpenMP code. This assumption should be verified.
148
149 Speaking with Jerome, it seems that using openMP and hdf5 can cause
150 some troubles. bitshuffle openMP version cause a lot of slowdown on
151 beamlines since it create an OMP scheduler per thread an this
152 invalidate the cache, producing very pour performances.
153
154 But it seems that in my case there is a dead lock. I do not know
155 for now how to determine what is the cause of this lock. (hdf5 ?).
156
157 I rebuilt bitstream without openMP and with hdf5-serial. now it
158 works. It would be nice to confirm the reactivating openMP, cause
159 some trouble.
160 ** TODO backport all this on buster.
161 * embedded libraries
162 All the un-bundled libraries are listed in the d/copyright file
163 under the Files-Excluded section.
164 ** DONE boost
165 Files-Excluded: build_dials/modules/boost
166 ** DONE mmdb
167 Files-Excluded: build_dials/modules/ccp4io/mmdb
168 ** DONE ssm
169 Files-Excluded: build_dials/modules/ccp4io/ssm
170 ** DONE antlr3
171 Files-Excluded: build_dials/modules/cctbx_project/ucif/antlr3
172 ** DONE clipper
173 Files-Excluded: build_dials/modules/clipper
174 ** DONE eiger
175 Files-Excluded: build_dials/modules/eigen
176 ** DONE gl2ps
177 Files-Excluded: build_dials/modules/gui_resources/gl2ps
178 ** DONE msgpack
179 Files-Excluded: build_dials/modules/msgpack-3.1.1
180 ** DONE scons
181 Files-Excluded: build_dials/modules/scons
182 ** DONE tntbx (header only library)
183 Files-Excluded: build_dials/modules/tntbx/include
184 ** TODO ccp4
185 The sources of this library is patched during the build process.
186 So we can not use the system one for now.
187 ** TODO annlib
188 Same thing, two version of the library are used, on with a variable
189 set to TRUE and the another one with FALSE. the namespace of the
190 second library is changed to avoid symbol collision.
191 ** TODO pycbf
192 - [X] packaging de python-pycbf #681997 -> get rid of the cbflib part.
193 - [ ] The pycbf embedded is compiled with
194 SWIG_PYTHON_STRICT_BYTE_CHAR which is not the case of the official
195 pycbf. Their is two way to deal with this. recompile the cbflib
196 with this option, or fix the dials code to provide normal python3
197 string instead of bytes. In fact the problem seems to be with the
198 filepath part. On unix filepath are in reality bytes. so the file
199 patch should be considere bytes but not the other strings. to my
200 opinin, the python strings should not be considere strict byte all
201 the time. only path on linux system should. this is why types are
202 good when programming (it not enough, right types is important).
203 https://github.com/cctbx/cctbx_project/issues/475
204 ** TODO gui_resources
205 need to check if the icons are specifiques or generic from the crystal_project
206 ** TODO javascript minified libraries
207 - [ ] E: dials source: source-is-missing modules/cctbx_project/crys3d/hklview/ngl.js line length is 32768 characters (>512)
208 - [ ] E: dials source: source-is-missing modules/dials/static/js/MathJax.js line length is 32123 characters (>512)
209 - [ ] E: dials source: source-is-missing modules/dials/static/js/bootstrap.min.js
210 - [ ] E: dials source: source-is-missing modules/dials/static/js/jquery-1.12.0.min.js
211 - [ ] E: dials source: source-is-missing modules/dials/static/js/plotly-latest.min.js
212 - [ ] E: dials source: source-is-missing modules/dials/static/katex/contrib/auto-render.min.js
213 - [ ] E: dials source: source-is-missing modules/dials/static/katex/katex.min.js
214 * tests
215 ** pytest
216 - pytest, missing extension.
217
218 ImportError while importing test module
219 '/usr/lib/python3/dist-packages/cctbx/examples/merging/test_cc_half.py'.
220 Hint: make sure your test modules/packages have valid Python
221 names.
222 Traceback:
223 /usr/lib/python3/dist-packages/boost/python.py:31: in import_ext
224 try: mod = __import__(name)
225 E ModuleNotFoundError: No module named 'scitbx_examples_bevington_ext'
226
227 During handling of the above exception, another exception occurred:
228 /usr/lib/python3/dist-packages/cctbx/examples/merging/__init__.py:2: in <module>
229 from scitbx.examples import bevington # import dependency
230 /usr/lib/python3/dist-packages/scitbx/examples/bevington/__init__.py:4: in <module>
231 ext = boost.python.import_ext("scitbx_examples_bevington_ext")
232 /usr/lib/python3/dist-packages/boost/python.py:40: in import_ext
233 raise ImportError(
234 E ImportError: __import__("scitbx_examples_bevington_ext"): No module named 'scitbx_examples_bevington_ext'
235 E sys.path:
236 E /usr/lib/python3/dist-packages
237 E /usr/bin
238 E /usr/lib/python38.zip
239 E /usr/lib/python3.8
240 E /usr/lib/python3.8/lib-dynload
241 E /usr/local/lib/python3.8/dist-packages
242 E /usr/lib/python3/dist-packages
243 E /usr/lib/python3.8/dist-packages
244
245 this extension is missing due to the removal of eigen. scitbx_examples_bevington_ext
246 ** packaging
247 - packaging dials-data pypi, maybe better if integrated in the git repository.
248 - packaging de chimera http://www.cgl.ucsf.edu/chimera/
249 ** runtimeError
250 *** RuntimeError: detector library not found
251 ______________________________________________________________ ERROR collecting algorithms/indexing/test_assign_indices.py ______________________________________________________________
252 algorithms/indexing/test_assign_indices.py:25: in <module>
253 from dxtbx.model.experiment_list import Experiment, ExperimentList
254 ../dxtbx/model/experiment_list.py:35: in <module>
255 from dxtbx.serialize import xds
256 ../dxtbx/serialize/xds.py:16: in <module>
257 from dxtbx.model.detector_helpers_types import detector_helpers_types
258 ../dxtbx/model/detector_helpers_types.py:87: in <module>
259 detector_helpers_types = detector_helpers_types()
260 ../dxtbx/model/detector_helpers_types.py:29: in __init__
261 raise RuntimeError("detector library not found")
262 E RuntimeError: detector library not found
263
264
265 We need to install also the data of dials.
266 Here the dxtbx detector library locvated here
267 dxtbx/data/detectors.lib
268
269 *** AssertionError: No models registered with dxtbx.scaling_model_ext entry point
270 ___________________________________________________________________ ERROR collecting algorithms/scaling/test_model.py ___________________________________________________________________
271 algorithms/scaling/test_model.py:10: in <module>
272 from dials.algorithms.scaling.model.model import (
273 algorithms/scaling/model/model.py:973: in <module>
274 assert (
275 E AssertionError: No models registered with dxtbx.scaling_model_ext entry point
276 test/command_line/test_cosym.py:10: in <module>
277 from dials.algorithms.symmetry.cosym._generate_test_data import (
278 algorithms/symmetry/__init__.py:24: in <module>
279 from dials.util.resolutionizer import Resolutionizer, phil_defaults
280 util/resolutionizer.py:12: in <module>
281 from dials.algorithms.scaling.scaling_library import determine_best_unit_cell
282 algorithms/scaling/scaling_library.py:25: in <module>
283 from dials.algorithms.scaling.model.model import KBScalingModel
284 algorithms/scaling/model/model.py:973: in <module>
285 assert (
286 E AssertionError: No models registered with dxtbx.scaling_model_ext entry point
287
288
289 this code is triggered by
290
291 model_phil_scope = phil.parse("")
292 _dxtbx_scaling_models = {
293 ep.name: ep for ep in pkg_resources.iter_entry_points("dxtbx.scaling_model_ext")
294 }
295 assert (
296 _dxtbx_scaling_models
297 ), "No models registered with dxtbx.scaling_model_ext entry point"
298
299
300 so we need to find what are the entry point of the dxtbx.scaling_model_ext.
301
302 These entry_points are stored in the egg files, which are created by
303 the build system. Is it a good idea to write a setup.py file which
304 take care of this.
305
306 I found three of them in the original build directory:
307
308 $ find . -name *egg*
309 ./lib/libtbx.scitbx.egg-info
310 ./lib/libtbx.dxtbx.egg-info
311 ./lib/libtbx.dials.egg-info
312
313
314 * for upstream
315 ** TODO os.environ[FLAG]
316 some code use os.environ[FLAG] but does not guard agains the
317 KeyError exception. It is better to use os.environ.get(FLAG) which
318 return None instead of throwing an exception.