Debian Codegraph @

docs/src/cdist-real-world.rst

raw
1 Dive into real world cdist
2 ==========================
3
4 Introduction
5 ------------
6
7 This walkthrough shows real world cdist configuration example.
8
9 Sample target host is named **test.ungleich.ch**.
10 Just replace **test.ungleich.ch** with your target hostname.
11
12 Our goal is to configure python application hosting. For writing sample
13 application we will use `Bottle <http://bottlepy.org>`_ WSGI micro web-framework.
14 It will use PostgreSQL database and it will list items from **items** table.
15 It will be served by uWSGI server. We will also use the Nginx web server
16 as a reverse proxy and we want HTTPS.
17 For HTTPS we will use Let's Encrypt certificate.
18
19 For setting up hosting we want to use cdist so we will write a new type
20 for that. This type will:
21
22 - install required packages
23 - create OS user, user home directory and application home directory
24 - create PostgreSQL database
25 - configure uWSGI
26 - configure Let's Encrypt certificate
27 - configure nginx.
28
29 Our type will not create the actual python application. Its intention is only
30 to configure hosting for specified user and project. It is up to the user to
31 create his/her applications.
32
33 So let's start.
34
35 Creating type layout
36 --------------------
37
38 We will create a new custom type. Let's call it **__sample_bottle_hosting**.
39
40 Go to **~/.cdist/type** directory (create it if it does not exist) and create
41 new type layout::
42
43 cd ~/.cdist/type
44 mkdir __sample_bottle_hosting
45 cd __sample_bottle_hosting
46 touch manifest gencode-remote
47 mkdir parameter
48 touch parameter/required
49
50 Creating __sample_bottle_hosting type parameters
51 ------------------------------------------------
52
53 Our type will be configurable through the means of parameters. Let's define
54 the following parameters:
55
56 projectname
57 name for the project, needed for uWSGI ini file
58
59 user
60 user name
61
62 domain
63 target host domain, needed for Let's Encrypt certificate.
64
65 We define parameters to make our type reusable for different projects, user and domain.
66
67 Define required parameters::
68
69 printf "projectname\n" >> parameter/required
70 printf "user\n" >> parameter/required
71 printf "domain\n" >> parameter/required
72
73 For details on type parameters see `Defining parameters <cdist-type.html#defining-parameters>`_.
74
75 Creating __sample_bottle_hosting type manifest
76 ----------------------------------------------
77
78 Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest).
79 We also want our type to currently support only Devuan. So we will start by
80 checking target host OS. We will use `os <cdist-reference.html#explorers>`_
81 global explorer::
82
83 os=$(cat "$__global/explorer/os")
84
85 case "$os" in
86 devuan)
87 :
88 ;;
89 *)
90 echo "OS $os currently not supported" >&2
91 exit 1
92 ;;
93 esac
94
95 If target host OS is not Devuan then we print error message to stderr
96 and exit. For other OS-es support we should check and change package names
97 we should install, because packages differ in different OS-es and in different
98 OS distributions like GNU/Linux distributions. There can also be a different
99 configuration locations (e.g. nginx config directory could be in /usr/local tree).
100 If we detected unsupported OS we should error out. cdist will stop configuration
101 process and output error message.
102
103 Creating user and user directories
104 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
105
106 Then we create user and his/her home directory and application home directory.
107 We will use existing cdist types `__user <man7/cdist-type__user.html>`_ and `__directory <man7/cdist-type__directory.html>`_::
108
109 user="$(cat "$__object/parameter/user")"
110 home="/home/$user"
111 apphome="$home/app"
112
113 # create user
114 __user "$user" --home "$home" --shell /bin/bash
115 # create user home dir
116 require="__user/$user" __directory "$home" \
117 --owner "$user" --group "$user" --mode 0755
118 # create app home dir
119 require="__user/$user __directory/$home" __directory "$apphome" \
120 --state present --owner "$user" --group "$user" --mode 0755
121
122 First we define *user*, *home* and *apphome* variables. User is defined by type's
123 **user** parameter. Here we use **require** which is cdist's way to define dependencies.
124 User home directory should be created **after** user is created. And application
125 home directory is created **after** both user and user home directory are created.
126 For details on **require** see `Dependencies <cdist-manifest.html#dependencies>`_.
127
128 Installing packages
129 ~~~~~~~~~~~~~~~~~~~
130
131 Install required packages using existing `__package <man7/cdist-type__package.html>`_ type.
132 Before installing package we want to update apt package index using
133 `__apt_update_index <man7/cdist-type__apt_update_index.html>`_::
134
135 # define packages that need to be installed
136 packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
137
138 # update package index
139 __apt_update_index
140 # install packages
141 for package in $packages_to_install
142 do require="__apt_update_index" __package $package --state=present
143 done
144
145 Here we use shell for loop. It executes **require="__apt_update_index" __package**
146 for each member in a list we define in **packages_to_install** variable.
147 This is much nicer then having as many **require="__apt_update_index" __package**
148 lines as there are packages we want to install.
149
150 For python packages we use `__package_pip <man7/cdist-type__package_pip.html>`_::
151
152 # install pip3 packages
153 for package in bottle bottle-pgsql; do
154 __package_pip --pip pip3 $package
155 done
156
157 Creating PostgreSQL database
158 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
159
160 Create PostgreSQL database using `__postgres_database <man7/cdist-type__postgres_database.html>`_
161 and `__postgres_role <man7/cdist-type__postgres_role.html>`_ for creating database user::
162
163 #PostgreSQL db & user
164 postgres_server=postgresql
165
166 # create PostgreSQL db user
167 require="__package/postgresql" __postgres_role $user --login --createdb
168 # create PostgreSQL db
169 require="__postgres_role/$user __package/postgresql" __postgres_database $user \
170 --owner $user
171
172 Configuring uWSGI
173 ~~~~~~~~~~~~~~~~~
174
175 Configure uWSGI using `__file <man7/cdist-type__file.html>`_ type::
176
177 # configure uWSGI
178 projectname="$(cat "$__object/parameter/projectname")"
179 require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
180 --owner root --group root --mode 0644 \
181 --state present \
182 --source - << EOF
183 [uwsgi]
184 socket = $apphome/uwsgi.sock
185 chdir = $apphome
186 wsgi-file = $projectname/wsgi.py
187 touch-reload = $projectname/wsgi.py
188 processes = 4
189 threads = 2
190 chmod-socket = 666
191 daemonize=true
192 vacuum = true
193 uid = $user
194 gid = $user
195 EOF
196
197 We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$user.ini** file.
198 Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**.
199 If this directory does not exist then **__file** type would error.
200 We also use stdin as file content source. For details see `Input from stdin <cdist-type.html#input-from-stdin>`_.
201 For feeding stdin we use here-document (**<<** operator). It allows redirection of subsequent
202 lines read by the shell to the input of a command until a line containing only the delimiter
203 and a newline, with no blank characters in between (EOF in our case).
204
205 Configuring nginx for Let's Encrypt and HTTPS redirection
206 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
207
208 Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this
209 purpose we will create new type **__sample_nginx_http_letsencrypt_and_ssl_redirect**
210 and use it here::
211
212 domain="$(cat "$__object/parameter/domain")"
213 webroot="/var/www/html"
214 __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
215
216 Configuring certificate creation
217 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
218
219 After HTTP nginx configuration we will create Let's Encrypt certificate using
220 `__letsencrypt_cert <man7/cdist-type__letsencrypt_cert.html>`_ type.
221 For Let's Encrypt cert configuration ensure that there is a DNS entry for your
222 domain. We assure that cert creation is applied after nginx HTTP is configured
223 for Let's Encrypt to work::
224
225 # create SSL cert
226 require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
227 __letsencrypt_cert --admin-email [email protected] \
228 --webroot "$webroot" \
229 --automatic-renewal \
230 --renew-hook "service nginx reload" \
231 --domain "$domain" \
232 "$domain"
233
234 Configuring nginx HTTPS server with uWSGI upstream
235 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
236
237 Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate::
238
239 # configure nginx
240 require="__package/nginx __letsencrypt_cert/$domain" \
241 __file "/etc/nginx/sites-enabled/https-$domain" \
242 --source - --mode 0644 << EOF
243 upstream _bottle {
244 server unix:$apphome/uwsgi.sock;
245 }
246
247 server {
248 listen 443;
249 listen [::]:443;
250
251 server_name $domain;
252
253 access_log /var/log/nginx/access.log;
254
255 ssl on;
256 ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem;
257 ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem;
258
259 client_max_body_size 256m;
260
261 location / {
262 try_files \$uri @uwsgi;
263 }
264
265 location @uwsgi {
266 include uwsgi_params;
267 uwsgi_pass _bottle;
268 }
269 }
270 EOF
271
272 Now our manifest is finished.
273
274 Complete __sample_bottle_hosting type manifest listing
275 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
276
277 Here is complete __sample_bottle_hosting type manifest listing,
278 located in ~/.cdist/type/__sample_bottle_hosting/manifest::
279
280 #!/bin/sh
281
282 os=$(cat "$__global/explorer/os")
283
284 case "$os" in
285 devuan)
286 :
287 ;;
288 *)
289 echo "OS $os currently not supported" >&2
290 exit 1
291 ;;
292 esac
293
294 projectname="$(cat "$__object/parameter/projectname")"
295 user="$(cat "$__object/parameter/user")"
296 home="/home/$user"
297 apphome="$home/app"
298 domain="$(cat "$__object/parameter/domain")"
299
300 # create user
301 __user "$user" --home "$home" --shell /bin/bash
302 # create user home dir
303 require="__user/$user" __directory "$home" \
304 --owner "$user" --group "$user" --mode 0755
305 # create app home dir
306 require="__user/$user __directory/$home" __directory "$apphome" \
307 --state present --owner "$user" --group "$user" --mode 0755
308
309 # define packages that need to be installed
310 packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
311
312 # update package index
313 __apt_update_index
314 # install packages
315 for package in $packages_to_install
316 do require="__apt_update_index" __package $package --state=present
317 done
318 # install pip3 packages
319 for package in bottle bottle-pgsql; do
320 __package_pip --pip pip3 $package
321 done
322
323 #PostgreSQL db & user
324 postgres_server=postgresql
325
326 # create PostgreSQL db user
327 require="__package/postgresql" __postgres_role $user --login --createdb
328 # create PostgreSQL db
329 require="__postgres_role/$user __package/postgresql" __postgres_database $user \
330 --owner $user
331 # configure uWSGI
332 require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
333 --owner root --group root --mode 0644 \
334 --state present \
335 --source - << EOF
336 [uwsgi]
337 socket = $apphome/uwsgi.sock
338 chdir = $apphome
339 wsgi-file = $projectname/wsgi.py
340 touch-reload = $projectname/wsgi.py
341 processes = 4
342 threads = 2
343 chmod-socket = 666
344 daemonize=true
345 vacuum = true
346 uid = $user
347 gid = $user
348 EOF
349
350 # setup nginx HTTP for Let's Encrypt and SSL redirect
351 domain="$(cat "$__object/parameter/domain")"
352 webroot="/var/www/html"
353 __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
354
355 # create SSL cert
356 require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
357 __letsencrypt_cert --admin-email [email protected] \
358 --webroot "$webroot" \
359 --automatic-renewal \
360 --renew-hook "service nginx reload" \
361 --domain "$domain" \
362 "$domain"
363
364 # configure nginx
365 require="__package/nginx __letsencrypt_cert/$domain" \
366 __file "/etc/nginx/sites-enabled/https-$domain" \
367 --source - --mode 0644 << EOF
368 upstream _bottle {
369 server unix:$apphome/uwsgi.sock;
370 }
371
372 server {
373 listen 443;
374 listen [::]:443;
375
376 server_name $domain;
377
378 access_log /var/log/nginx/access.log;
379
380 ssl on;
381 ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem;
382 ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem;
383
384 client_max_body_size 256m;
385
386 location / {
387 try_files \$uri @uwsgi;
388 }
389
390 location @uwsgi {
391 include uwsgi_params;
392 uwsgi_pass _bottle;
393 }
394 }
395 EOF
396
397 Creating __sample_bottle_hosting type gencode-remote
398 ----------------------------------------------------
399
400 Now define **gencode-remote** script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote.
401 After manifest is applied it should restart uWSGI and nginx services so that our
402 configuration is active. Our gencode-remote looks like the following::
403
404 echo "service uwsgi restart"
405 echo "service nginx restart"
406
407 Our **__sample_bottle_hosting** type is now finished.
408
409 Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type
410 --------------------------------------------------------------
411
412 Let's now create **__sample_nginx_http_letsencrypt_and_ssl_redirect** type::
413
414 cd ~/.cdist/type
415 mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect
416 cd __sample_nginx_http_letsencrypt_and_ssl_redirect
417 mkdir parameter
418 echo webroot > parameter/required
419 touch manifest
420 touch gencode-remote
421
422 Edit manifest::
423
424 domain="$__object_id"
425 webroot="$(cat "$__object/parameter/webroot")"
426 # make sure we have nginx package
427 __package nginx
428 # setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS
429 require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \
430 --source - --mode 0644 << EOF
431 server {
432 listen *:80;
433 listen [::]:80;
434
435 server_name $domain;
436
437 # Let's Encrypt
438 location /.well-known/acme-challenge/ {
439 root $webroot;
440 }
441
442 # Everything else -> SSL
443 location / {
444 return 301 https://\$host\$request_uri;
445 }
446 }
447
448 EOF
449
450 Edit gencode-remote::
451
452 echo "service nginx reload"
453
454 Creating init manifest
455 ----------------------
456
457 Next create init manifest::
458
459 cd ~/.cdist/manifest
460 printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample
461
462 Using this init manifest our target host will be configured using our **__sample_bottle_hosting**
463 type with projectname *sample*, user *app* and domain equal to **__target_host**.
464 Here the last positional argument *sample* is type's object id. For details on
465 **__target_host** and **__object_id** see
466 `Environment variables (for reading) <cdist-reference.html#environment-variables-for-reading>`_
467 reference.
468
469 Configuring host
470 ----------------
471
472 Finally configure test.ungleich.ch::
473
474 cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch
475
476 After cdist configuration is successfully finished our host is ready.
477
478 Creating python bottle application
479 ----------------------------------
480
481 We now need to create Bottle application. As you remember from the beginning
482 of this walkthrough our type does not create the actual python application,
483 its intention is only to configure hosting for specified user and project.
484 It is up to the user to create his/her applications.
485
486 Become app user::
487
488 su -l app
489
490 Preparing database
491 ~~~~~~~~~~~~~~~~~~
492
493 We need to prepare database for our application. Create table and
494 insert some items::
495
496 psql -c "create table items (item varchar(255));"
497
498 psql -c "insert into items(item) values('spam');"
499 psql -c "insert into items(item) values('eggs');"
500 psql -c "insert into items(item) values('sausage');"
501
502 Creating application
503 ~~~~~~~~~~~~~~~~~~~~
504
505 Next create sample app::
506
507 cd /home/app/app
508 mkdir sample
509 cd sample
510
511 Create app.py with the following content::
512
513 #!/usr/bin/env python3
514
515 import bottle
516 import bottle_pgsql
517
518 app = application = bottle.Bottle()
519 plugin = bottle_pgsql.Plugin('dbname=app user=app password=')
520 app.install(plugin)
521
522 @app.route('/')
523 def show_index(db):
524 db.execute('select * from items')
525 items = db.fetchall() or []
526 rv = '<html><body><h3>Items:</h3><ul>'
527 for item in items:
528 rv += '<li>' + str(item['item']) + '</li>'
529 rv += '</ul></body></html>'
530 return rv
531
532 if __name__ == '__main__':
533 bottle.run(app=app, host='0.0.0.0', port=8080)
534
535 Create wsgi.py with the following content::
536
537 import os
538
539 os.chdir(os.path.dirname(__file__))
540
541 import app
542 application = app.app
543
544 We have configured uWSGI with **touch-reload = $projectname/wsgi.py** so after
545 we have changed our **wsgi.py** file uWSGI reloads the application.
546
547 Our application selects and lists items from **items** table.
548
549 Opening application
550 ~~~~~~~~~~~~~~~~~~~~
551
552 Finally try the application::
553
554 http://test.ungleich.ch/
555
556 It should redirect to HTTPS and return:
557
558 .. container:: highlight
559
560 .. raw:: html
561
562 <h3>Items:</h3>
563
564 <ul>
565 <li>spam</li>
566 <li>eggs</li>
567 <li>sausage</li>
568 </ul>
569
570 What's next?
571 ------------
572
573 Continue reading next sections ;)