Update technical documentation
This commit is contained in:
parent
615b847cf9
commit
40e98ff42a
2
apk/spoc
2
apk/spoc
@ -1 +1 @@
|
||||
Subproject commit 021bac46b39ef25b0daea1eb6c4ee0c38160a89f
|
||||
Subproject commit 7b3c2088e12516a5f1901f92db68602f911f1245
|
||||
@ -10,7 +10,6 @@ apk add git file htop less openssh-client tree
|
||||
apk add alpine-sdk
|
||||
# Install Sphinx support
|
||||
apk add py3-sphinx
|
||||
pip3 install recommonmark sphinx-markdown-tables
|
||||
|
||||
# Copy root profile files and settings
|
||||
mkdir -p /root/.config/htop
|
||||
|
||||
78
doc/applications/ckan.rst
Normal file
78
doc/applications/ckan.rst
Normal file
@ -0,0 +1,78 @@
|
||||
CKAN
|
||||
====
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It is written in python, has Flask-based web interface and uses Postgres database with PostGIS extension. CKAN exposes API which allows third parties to create extensions.
|
||||
|
||||
The project is split to several components. The most important ones are:
|
||||
|
||||
- **ckan** - https://github.com/ckan/ckan - The main DMS component with Flask frontend.
|
||||
- **datapusher** - https://github.com/ckan/datapusher - A standalone web service that pushes data files from a CKAN site's resources into its DataStore.
|
||||
- **ckan-service-provider** - https://github.com/ckan/ckan-service-provider - A library for making web services that make functions available as synchronous or asynchronous jobs.
|
||||
|
||||
Apart from that, CKAN requires a *Redis* in-memory data store and *Solr* search platform instance. Note that CKAN is shipped with Solr schemas which don't work well with up-to-date versions of Solr, therefore we use older Solr 6.5.1.
|
||||
|
||||
DataPusher
|
||||
----------
|
||||
|
||||
DataPusher is running as a separate service. Its development is a bit out of sync with the core ckan component. SKAN has already moved to python 3, however datapusher is still working only with python 2.7. Due to that, DataPusher has its own image and is running as a separate container.
|
||||
|
||||
Alpine, being a progressive linux distribution, stopped supporting uwsgi connector for python 2.7 in Alpine 3.11, therefore the container needs to be based on older Alpine 3.10 which still ships with *uwsgi-python2*.
|
||||
|
||||
The connection between CKAN and DataPusher is done via HTTPS endpoints. There is a possibility to disable TLS certificate verification, but a better solution seems to be import any self-signed certificate into the DataPusher's trust store. There is a script which automatically handles the import whenever the DataPusher container starts.
|
||||
|
||||
Extensions
|
||||
----------
|
||||
|
||||
CKAN container comes with several preinstalled and preconfigured extensions. Historically, there were more extensions, but over the time CKAN's API evolved and the development has moved to python 3, so majority of the historical extensions are no longer operational.
|
||||
|
||||
ckan/ckanext-basiccharts
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/ckan/ckanext-basiccharts
|
||||
|
||||
Adds Line, Bar and Pie charts. It uses Flot Charts, which is compatible with all major browsers (including IE6+).
|
||||
|
||||
ckan/ckanext-spatial
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/ckan/ckanext-spatial
|
||||
|
||||
Contains plugins that add geospatial capabilities to CKAN. E.g. a spatial field on the default CKAN dataset schema which allows to perform spatial queries and to display the dataset extent on the frontend.
|
||||
|
||||
ckan/ckanext-geoview
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/ckan/ckanext-geoview
|
||||
|
||||
Contains view plugins to display geospatial files and services in CKAN. It contains an OpenLayers based viewer and other view plugins that used to be part of ckanext-spatial.
|
||||
|
||||
ckan/ckanext-mapviews
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/ckan/ckanext-mapviews
|
||||
|
||||
Adds regular and choropleth maps to CKAN, using the new CKAN Resource View. It uses LeafletJS, which is compatible with all major browsers (including IE7+).
|
||||
|
||||
XVTSolutions/ckanext-spatialUI
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/XVTSolutions/ckanext-spatialUI
|
||||
|
||||
Provides the UI elements for spatial search. Spatial search widget and dataset extent map.
|
||||
|
||||
aptivate/ckanext-datasetthumbnail
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/aptivate/ckanext-datasetthumbnail
|
||||
|
||||
Adds support for generation and display of thumbnail images.
|
||||
|
||||
datagvat/ckanext-dgvat_xls
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/datagvat/ckanext-dgvat_xls
|
||||
|
||||
Allows group admin to export datasets from their organizations to export all datasets to a .xls worksheet. Sysadmins may choose to export datasets from any organization they want or to export all datasets of all groups.
|
||||
21
doc/applications/images-list-apps.csv
Normal file
21
doc/applications/images-list-apps.csv
Normal file
@ -0,0 +1,21 @@
|
||||
Application,Container,Host
|
||||
CKAN,ckan,ckan
|
||||
Crisis Cleanup,crisiscleanup,cc
|
||||
CTS,cts,cts
|
||||
EcoGIS,ecogis,ecogis
|
||||
FrontlineSMS,frontlinesms,sms
|
||||
GNU Health,gnuhealth,gh
|
||||
Kanboard,kanboard,kb
|
||||
Mifos X,mifosx,mifosx
|
||||
Motech,motech,motech
|
||||
ODK Aggregate,opendatakit,odk
|
||||
ODK Build,opendatakit-build,odkbuild
|
||||
Odoo,odoo,odoo
|
||||
OpenMapKit,openmapkit,omk
|
||||
Pan.do/ra,pandora,pandora
|
||||
Sahana,sahana,sahana
|
||||
Sahana - Demo,sahana-demo,sahana-demo
|
||||
SAMBRO,sambro,sambro
|
||||
SeedDMS,seeddms,dms
|
||||
Sigmah,sigmah,sigmah
|
||||
Ushahidi,ushahidi,ush
|
||||
|
13
doc/applications/images-list-basic.csv
Normal file
13
doc/applications/images-list-basic.csv
Normal file
@ -0,0 +1,13 @@
|
||||
Product,Layer
|
||||
Alpine 3.8,alpine3.8
|
||||
Alpine 3.8 - PHP 5.6,alpine3.8-php5.6
|
||||
Alpine 3.9 - Ruby 2.4,alpine3.8-ruby2.4
|
||||
Alpine 3.9,alpine3.9
|
||||
Alpine 3.9 - Java 8,alpine3.9-java8
|
||||
Alpine 3.9 - PHP 7.2,alpine3.9-php7.2
|
||||
Alpine 3.9 - Python 2.7,alpine3.9-python2.7
|
||||
Alpine 3.9 - Python 3.6,alpine3.9-python3.6
|
||||
Alpine 3.9 - Ruby 2.4,alpine3.9-ruby2.4
|
||||
Alpine 3.9 - Ruby 2.6,alpine3.9-ruby2.6
|
||||
Alpine 3.9 - Tomcat 7,alpine3.9-tomcat7
|
||||
Alpine 3.9 - Tomcat 8.5,alpine3.9-tomcat8.5
|
||||
|
8
doc/applications/images-list-services.csv
Normal file
8
doc/applications/images-list-services.csv
Normal file
@ -0,0 +1,8 @@
|
||||
Product,Layer,UID/GID,Internal Port
|
||||
ActiveMQ,activemq,61616,61616 (ActiveMQ)
|
||||
MariaDB,mariadb,3306,3306 (MySQL)
|
||||
Postgres,postgres,5432,5432 (Postgres)
|
||||
PostGIS,postgis,5432,5432 (Postgres)
|
||||
RabbitMQ,rabbitmq,5672,5672 (AMQP)
|
||||
Redis,redis,6379,6379 (Redis)
|
||||
Solr 6,solr6,8983,8983 (HTTP)
|
||||
|
25
doc/applications/images-list.rst
Normal file
25
doc/applications/images-list.rst
Normal file
@ -0,0 +1,25 @@
|
||||
List of existing container images
|
||||
=================================
|
||||
|
||||
Basic and runtime images
|
||||
------------------------
|
||||
|
||||
.. csv-table::
|
||||
:file: images-list-basic.csv
|
||||
:header-rows: 1
|
||||
|
||||
Common service images
|
||||
---------------------
|
||||
|
||||
.. csv-table::
|
||||
:file: images-list-services.csv
|
||||
:header-rows: 1
|
||||
|
||||
List of application images
|
||||
--------------------------
|
||||
|
||||
All application images have the application user UID/GID 8080 and listen on internal port 8080 (HTTP).
|
||||
|
||||
.. csv-table::
|
||||
:file: images-list-apps.csv
|
||||
:header-rows: 1
|
||||
14
doc/applications/index.rst
Normal file
14
doc/applications/index.rst
Normal file
@ -0,0 +1,14 @@
|
||||
Applications and containers
|
||||
===========================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
images-list
|
||||
tech-knowledge
|
||||
map-services
|
||||
ckan
|
||||
kanboard
|
||||
sahana
|
||||
sahana-configuration-report
|
||||
seeddms
|
||||
121
doc/applications/kanboard.rst
Normal file
121
doc/applications/kanboard.rst
Normal file
@ -0,0 +1,121 @@
|
||||
Kanboard
|
||||
========
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Kanboard is a Kanban project management tool written in PHP. It supports creation of projects and their population with tasks, provides Kanban boards with swimlanes, charts etc. It is extensile via plugins.
|
||||
|
||||
Upstream URL: https://github.com/kanboard/kanboard
|
||||
|
||||
Localization
|
||||
------------
|
||||
|
||||
Kanboard is localized via PHP files containing associative arrays with original English string and the translation in the respective language. Every plugin has its own localization files. Due to this, we currently maintain forks or 4 repositories with plugins where the original maintainer decided to discontinue support.
|
||||
|
||||
Plugins
|
||||
-------
|
||||
|
||||
Following list summarizes state of Kanboard plugins used or requested in our container as of 05/2020.
|
||||
|
||||
BlueTeck/kanboard_plugin_overwrite_translation
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/BlueTeck/kanboard_plugin_overwrite_translation
|
||||
|
||||
Supplies translation strings via additional configuration file for other plugins which are missing or have incorrect translations. Not updated anymore, however the maintainer is active on GitHub. This plugin has initially been requested to be included in our installation, but due to its purpose, it's mostly useless as all translations were done in the respective plugin localization files.
|
||||
|
||||
BlueTeck/kanboard_plugin_coverimage
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/BlueTeck/kanboard_plugin_coverimage
|
||||
|
||||
This plugin adds a cover image function to tasks on the board, and allows you to set a project image to use as a logo for projects. Development is active.
|
||||
|
||||
BlueTeck/kanboard_plugin_metadata
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/BlueTeck/kanboard_plugin_metadata
|
||||
|
||||
This plugin adds a basic GUI for Plugin Metadata. Not updated anymore, however the maintainer is active on GitHub.
|
||||
|
||||
eSkiSo/Subtaskdate
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/eSkiSo/Subtaskdate
|
||||
|
||||
Adds a new due date field for subtasks, including events, API and search functions. Development is active.
|
||||
|
||||
kanboard/plugin-budget
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-budget
|
||||
- Fork URL: https://github.com/trendspotter/kanboard-plugin-budget
|
||||
|
||||
Allows to create budget lines, see expenses based on sub-task time tracking and manage user hourly rates. Repository has been forked. The original repository is archived.
|
||||
|
||||
kanboard/plugin-calendar
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-calendar
|
||||
|
||||
Adds embedded calendar view for Kanboard. Development is active.
|
||||
|
||||
kanboard/plugin-chat
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-chat
|
||||
- Fork URL: https://github.com/trendspotter/kanboard-plugin-chat
|
||||
|
||||
Minimalist internal chat for Kanboard. Allows only one room for all users. Repository has been forked. The original repository is archived.
|
||||
|
||||
kanboard/plugin-gantt
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-gantt
|
||||
|
||||
Adds Gantt charts for Kanboard. Development is active.
|
||||
|
||||
kanboard/plugin-gravatar
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-gravatar
|
||||
|
||||
Enables Gravatar icons for Kanboard users. Repository is archived. There are no strings for translation, so it hasn't been forked.
|
||||
|
||||
kanboard/plugin-registration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-registration
|
||||
|
||||
Allows new user to sign up to Kanboard. Development is active.
|
||||
|
||||
kanboard/plugin-sms-2fa
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-sms-2fa
|
||||
- Fork URL: https://github.com/trendspotter/kanboard-plugin-sms-2fa
|
||||
|
||||
Allows to use text messages (SMS) instead of the default TOTP system (Time-based One-time Password Algorithm). Works with Twilio service. Repository has been forked. The original repository is archived.
|
||||
|
||||
kanboard/plugin-task-board-date
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/kanboard/plugin-task-board-date
|
||||
|
||||
Adds new date field for tasks to define the visibility on the board and dashboard. Repository is archived. There are no strings for translation, so it hasn't been forked.
|
||||
|
||||
oliviermaridat/kanboard-milestone-plugin
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/oliviermaridat/kanboard-milestone-plugin
|
||||
|
||||
Adds section for milestones to show their related tasks. Development is active.
|
||||
|
||||
xavividal/kanboard-plugin-relationgraph
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
- URL: https://github.com/xavividal/kanboard-plugin-relationgraph
|
||||
- Fork URL: https://github.com/trendspotter/kanboard-plugin-relationgraph
|
||||
|
||||
Shows relations between tasks using a graph library. Repository has been forked. The original repository is inactive.
|
||||
17
doc/applications/map-services.csv
Normal file
17
doc/applications/map-services.csv
Normal file
@ -0,0 +1,17 @@
|
||||
Application,Data sources,Map viewer,Configurable,Notes
|
||||
CKAN,OSM (`Stamen <http://maps.stamen.com>`_),Leaflet,No,[1]
|
||||
CrisisCleanup,Google Maps,Google Maps,No,
|
||||
CTS,"OSM, `ArcGIS <http://server.arcgisonline.com/arcgis/rest/services>`_",Leaflet,No,
|
||||
EcoGIS,,FreeGIS + OpenLayers 2,Probably,[2]
|
||||
Odoo,Google Maps,Google Maps,No,[3]
|
||||
OpenMapKit,OSM,N/A,Yes,[4]
|
||||
Pan.do/ra,Google Maps,Google Maps + OxMap,No,
|
||||
Sahana Eden,OSM (`HOT <https://www.hotosm.org/>`_),OpenLayers 2,"Yes, very",[5]
|
||||
Ushahidi,"OSM (`Mapbox <https://www.mapbox.com/about/maps/>`_, `HOT`_)",Leaflet,No,
|
||||
*---*,,,,
|
||||
WebODM,OSM,Leaflet,,
|
||||
Crismapp,OSM,Leaflet,,
|
||||
ThinkHazard!,OSM,Mapbox,,
|
||||
Openforis,Google Earth,,No,
|
||||
Tendenci,Google Maps,Google Maps,No,
|
||||
GeoNode,OSM,OpenLayers 3 (?),,
|
||||
|
12
doc/applications/map-services.rst
Normal file
12
doc/applications/map-services.rst
Normal file
@ -0,0 +1,12 @@
|
||||
Map services used in applications
|
||||
=================================
|
||||
|
||||
.. csv-table::
|
||||
:file: map-services.csv
|
||||
:header-rows: 1
|
||||
|
||||
1. Used by CKAN extensions *reclineview*, *spatial* and *geoview*.
|
||||
2. Untested as the EcoGIS source code is not fully open. Looks like the data sources are configurable, but the full documentation is only in Italian.
|
||||
3. Used by Odoo *Google Maps* module to display company/partner address on map.
|
||||
4. Map is used by OMK and ODK Android clients. OMK Server only offers the API. See `area of interest deployment <http://posm.io/docs/omk/walkthrough/>`_ on POSM wiki.
|
||||
5. Sahana Eden supports multitude of connectors and protocols to process map and feature data. ArcGIS REST, Bing maps, GeoJSON, GeoRSS, Google Maps, GPX, KML, MGRS, OSM, OWM, Shapefile, TMS, WFS, WMS and XYZ.
|
||||
1490
doc/applications/sahana-configuration-report.rst
Normal file
1490
doc/applications/sahana-configuration-report.rst
Normal file
File diff suppressed because it is too large
Load Diff
21
doc/applications/sahana.rst
Normal file
21
doc/applications/sahana.rst
Normal file
@ -0,0 +1,21 @@
|
||||
Sahana Eden
|
||||
===========
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Sahana Eden is a flexible and extensible humanitarian platform written in python, running on web2py platform and PostgreSQL with PostGIS extension. Sahana Eden is extremely highly customizable and configurable which unfortunately often leads to various bugs and misunderstandings in the configuration. For the description of the individual configuration options and their effect, see `Sahana configuration report <sahana-configuration-report>`_. The application is usually configured via templates, which contain all the configuration, forms and logic customizations and some basic data relevant for the intended deployment.
|
||||
|
||||
The application supports various interfaces, protocols and formats for data exchange - CAP (Common Alerting Protocol), XML/Atom feeds, WMS, WFS, WCS and other map services, KML exports, XML and CSV tabular data etc. Maps can be configured and adjusted to use virtually any tile, layer and feature sets. See `map services <map-services>`_.
|
||||
|
||||
Upstream URL: https://github.com/sahana/eden
|
||||
|
||||
Setup module
|
||||
------------
|
||||
|
||||
Sahana deployment functions are based on Ansible. Ansible is provisioning, configuration management, and application-deployment tool written in python. It uses YAML files to define the desired actions or states. These files are called *playbooks*. There is a whole separate sahana_deploy project focusing on rapid Sahana Eden deployment and parts of it are used also in the Setup module. The setup module allows to control Sahana instances and adjust their configuration in terms of setting simple values and enabling/disabling modules in ``000_config.py``. The module, as well as the Ansible playbooks for the whole deployment are a bit too strict when it comes to assumptions and discovery of Sahana's environment. E.g. the path to the instance is expected to be always under ``/home/prod``. The Setup module is enabled and can be used, however due to its limitations and expectations it sets, it can't be considered reliable in the environment of Alpine container.
|
||||
|
||||
Python 3.8 compatibility
|
||||
------------------------
|
||||
|
||||
Sahana Eden is currently recommended to be run on web2py R2.18.5 (commit hash `59700b8 <https://github.com/web2py/web2py/commit/59700b8>`_). Recommended version is tracked in eden_deploy project in `main.yml for common playbook <https://github.com/sahana/eden_deploy/blob/master/roles/common/tasks/main.yml>`_. This version however doesn't fully support python 3.8, so a series of patches needs to applied to achieve the full support for python 3.8. This includes compatibility patches for gluon and pydal tracked in `_compat.patch <https://github.com/sahana/eden_deploy/blob/master/roles/common/files/_compat.patch>`_ and scheduler patch tracked separately in `scheduler.diff <https://github.com/sahana/eden_deploy/blob/master/roles/common/files/scheduler.diff>`_.
|
||||
70
doc/applications/seeddms.rst
Normal file
70
doc/applications/seeddms.rst
Normal file
@ -0,0 +1,70 @@
|
||||
SeedDMS
|
||||
=======
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
SeedDMS is a free document management system with web based user interface. It is based on PHP and PostgreSQL and uses built-in *Lucene* indexing engine for fulltext search.
|
||||
|
||||
Upstream URL: https://sourceforge.net/projects/seeddms/
|
||||
|
||||
Previewers and converters
|
||||
-------------------------
|
||||
|
||||
SeedDMS uses LibreOffice components to generate thumbnails and PDFs of the imported documents. This is also useful for extraction of text for fulltext search. The main entry point is a third-party wrapper script called *unoconv* available from https://github.com/unoconv/unoconv/. This script wraps conversion functions of LibreOffice components, allowing to create a PDF out of virtually any ODF or OOXML document.
|
||||
|
||||
For previewers, additional chaining with *GhostScript* and *ImageMagick* is done in order to generate a thumbnail of the document. All conversions are entered as custom-defined commands in ``settings.xml`` SeedDMS configuration file.
|
||||
|
||||
.. code-block:: xml
|
||||
|
||||
<advanced>
|
||||
<converters target="fulltext">
|
||||
<converter mimeType="application/pdf">pdftotext -enc UTF-8 -nopgbrk %s -</converter>
|
||||
<converter mimeType="text/rtf">unoconv -d document -f txt --stdout %s</converter>
|
||||
<converter mimeType="application/msword">unoconv -d document -f txt --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.text">unoconv -d document -f txt --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.wordprocessingml.document">unoconv -d document -f txt --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.ms-excel">unoconv -d spreadsheet -f csv --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.spreadsheet">unoconv -d spreadsheet -f csv --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet">unoconv -d spreadsheet -f csv --stdout %s</converter>
|
||||
<converter mimeType="application/vnd.ms-powerpoint">unoconv -d presentation -f pdf --stdout %s | pdftotext -enc UTF-8 -nopgbrk - -</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.presentation">unoconv -d presentation -f pdf --stdout %s | pdftotext -enc UTF-8 -nopgbrk - -</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.presentationml.presentation">unoconv -d presentation -f pdf --stdout %s | pdftotext -enc UTF-8 -nopgbrk - -</converter>
|
||||
<converter mimeType="text/html">unoconv -d document -f txt --stdout %s</converter>
|
||||
<converter mimeType="text/plain">cat %s</converter>
|
||||
</converters>
|
||||
<converters target="preview">
|
||||
<converter mimeType="image/png">convert -resize %wx '%f' '%o'</converter>
|
||||
<converter mimeType="image/gif">convert -resize %wx '%f' '%o'</converter>
|
||||
<converter mimeType="image/jpg">convert -resize %wx '%f' '%o'</converter>
|
||||
<converter mimeType="image/jpeg">convert -resize %wx '%f' '%o'</converter>
|
||||
<converter mimeType="image/svg+xml">convert -resize %wx '%f' '%o'</converter>
|
||||
<converter mimeType="application/pdf">convert -density 100 -resize %wx '%f[0]' '%o'</converter>
|
||||
<converter mimeType="text/rtf">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/msword">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.text">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.wordprocessingml.document">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.ms-excel">unoconv -d spreadsheet -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.spreadsheet">unoconv -d spreadsheet -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet">unoconv -d spreadsheet -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.ms-powerpoint">unoconv -d presentation -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.presentation">unoconv -d presentation -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.presentationml.presentation">unoconv -d presentation -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="text/html">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
<converter mimeType="text/plain">unoconv -d document -e PageRange=1-1 -f pdf --stdout '%f' | gs -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dPDFFitPage -r72 -sOutputFile=- -q - | convert -resize %wx png:- '%o'</converter>
|
||||
</converters>
|
||||
<converters target="pdf">
|
||||
<converter mimeType="text/rtf">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/msword">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.text">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.wordprocessingml.document">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.ms-excel">unoconv -d spreadsheet -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.spreadsheet">unoconv -d spreadsheet -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet">unoconv -d spreadsheet -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.ms-powerpoint">unoconv -d presentation -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.oasis.opendocument.presentation">unoconv -d presentation -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="application/vnd.openxmlformats-officedocument.presentationml.presentation">unoconv -d presentation -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="text/html">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
<converter mimeType="text/plain">unoconv -d document -f pdf -o '%o' '%f'</converter>
|
||||
</converters>
|
||||
</advanced>
|
||||
@ -1,10 +1,13 @@
|
||||
# Overview of technological requirements
|
||||
Technological knowledge requirements
|
||||
====================================
|
||||
|
||||
## Basic system
|
||||
Basic system
|
||||
-------------
|
||||
|
||||
Components which are installed directly as part of the basic virtual machine.
|
||||
|
||||
### Alpine linux
|
||||
Alpine linux
|
||||
^^^^^^^^^^^^
|
||||
|
||||
- **General description:** Operating system
|
||||
- **Tech. description:** Lightweight linux distribution based on musl C libraries
|
||||
@ -12,7 +15,8 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** Everything
|
||||
- **Related skills:** EXTLINUX / ISOLINUX, git, linux administration (cron, filesystems, iptables, networking, user/group mgmt etc.), LUKS, LVM, OpenRC init system, POSIX standards, s6 init system, shell scripting
|
||||
|
||||
### Acme.sh
|
||||
Acme.sh
|
||||
^^^^^^^
|
||||
|
||||
- **General description:** Certificate renewal tool
|
||||
- **Tech. description:** Shell-based Automated Certificate Management Environment client
|
||||
@ -20,7 +24,8 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** nginx, VMMgr
|
||||
- **Related skills:** shell scripting, SSL/TLS
|
||||
|
||||
### Nginx
|
||||
Nginx
|
||||
^^^^^
|
||||
|
||||
- **General description:** Web server
|
||||
- **Tech. description:** Lightweight HTTP server
|
||||
@ -28,7 +33,8 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** All application containers, VMMgr
|
||||
- **Related skills:** HTTP (proxying, rewriting)
|
||||
|
||||
### LXC
|
||||
LXC
|
||||
^^^
|
||||
|
||||
- **General description:** Container virtualization host
|
||||
- **Tech. description:** Operating system-level container virtualization host
|
||||
@ -36,7 +42,8 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** All containers
|
||||
- **Related skills:** container virtualization fundamentals, linux kernel (cgroups, overlayfs, seccomp), shell scripting
|
||||
|
||||
### Postfix
|
||||
Postfix
|
||||
^^^^^^^
|
||||
|
||||
- **General description:** Mail server
|
||||
- **Tech. description:** Outbound mail transfer agent
|
||||
@ -44,7 +51,17 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** All application containers
|
||||
- **Related skills:** SMTP
|
||||
|
||||
### VMMgr
|
||||
SPOC
|
||||
^^^^
|
||||
|
||||
- **General description:** Container virtualization manager
|
||||
- **Tech. description:** Convenience abstraction layer on top of LXC
|
||||
- **Depends on:** LXC, Python 3
|
||||
- **Used by:** All containers, VMMgr
|
||||
- **Related skills:** JSON, python 3 frameworks and modules (cryptography, requests, subprocess), LXC, shell scripting
|
||||
|
||||
VMMgr
|
||||
^^^^^
|
||||
|
||||
- **General description:** Virtual machine and application manager web interface
|
||||
- **Tech. description:** In-house Werkzeug-based virtual machine and application manager WSGI application
|
||||
@ -52,12 +69,13 @@ Components which are installed directly as part of the basic virtual machine.
|
||||
- **Used by:** User
|
||||
- **Related skills:** JSON, python 3 frameworks and modules (cryptography, jinja2, requests, subprocess, werkzeug), shell scripting, WSGI application development
|
||||
|
||||
|
||||
## Runtimes
|
||||
Runtimes
|
||||
--------
|
||||
|
||||
Components which are supplied as LXC overlay layers but don't run as standalone containers
|
||||
|
||||
### Java
|
||||
Java
|
||||
^^^^
|
||||
|
||||
- **General description:** Java runtime environment
|
||||
- **Tech. description:** Java OpenJDK 8 runtime environment
|
||||
@ -65,39 +83,44 @@ Components which are supplied as LXC overlay layers but don't run as standalone
|
||||
- **Used by:** ActiveMQ, FrontlineSMS, Tomcat, CrisisCleanup, MifosX, Motech, OpenDataKit, OpenMapKit, Sigmah, Solr
|
||||
- **Related skills:** -
|
||||
|
||||
### Node.js
|
||||
Nodejs
|
||||
^^^^^^^
|
||||
|
||||
- **General description:** JavaScript runtime environment
|
||||
- **Tech. description:** Server-side Node.js 8 JavaScript runtime environment
|
||||
- **Tech. description:** Server-side Nodejs 8 JavaScript runtime environment
|
||||
- **Depends on:** -
|
||||
- **Used by:** CrisisCleanup, GNU Health, Odoo, OpenDataKit Build, OpenMapKit
|
||||
- **Related skills:** HTTP (proxying), JavaScript (language overview)
|
||||
|
||||
### PHP
|
||||
PHP
|
||||
^^^
|
||||
|
||||
- **General description:** PHP 7 runtime environment
|
||||
- **Tech. description:** PHP 7 hypertext preprocessor scripting runtime
|
||||
- **Depends on:** -
|
||||
- **Used by:** KanBoard, SeedDMS, Ushahidi
|
||||
- **Used by:** Kanboard, SeedDMS, Ushahidi
|
||||
- **Related skills:** HTTP (proxying), PHP 7 (language overview), PHP-FPM
|
||||
|
||||
### Python 2
|
||||
Python 2
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** Python 2 runtime environment
|
||||
- **Tech. description:** Python 2 runtime environemnt and standard libraries
|
||||
- **Tech. description:** Python 2 runtime environment and standard libraries
|
||||
- **Depends on:** -
|
||||
- **Used by:** CKAN, CKAN DataPusher, CTS, OpenMapKit, Sahana Eden
|
||||
- **Related skills:** linux compilation toolchain (header files, gcc, make etc.), pip, python (language overview)
|
||||
|
||||
### Python 3
|
||||
Python 3
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** Python 3 runtime environment
|
||||
- **Tech. description:** Python 3 runtime environemnt and standard libraries
|
||||
- **Tech. description:** Python 3 runtime environment and standard libraries
|
||||
- **Depends on:** -
|
||||
- **Used by:** GNU Health, Odoo, Pan.do/ra, SeedDMS, VMMgr (doesn't use container)
|
||||
- **Related skills:** linux compilation toolchain (header files, gcc, make etc.), pip, python (language overview)
|
||||
|
||||
### Ruby
|
||||
Ruby
|
||||
^^^^
|
||||
|
||||
- **General description:** Ruby runtime environment
|
||||
- **Tech. description:** Ruby 2.4 runtime environment and gem package installer
|
||||
@ -105,7 +128,8 @@ Components which are supplied as LXC overlay layers but don't run as standalone
|
||||
- **Used by:** CrisisCleanup, OpenDataKit Build
|
||||
- **Related skills:** linux compilation toolchain (header files, gcc, make etc.)
|
||||
|
||||
### Tomcat
|
||||
Tomcat
|
||||
^^^^^^
|
||||
|
||||
- **General description:** Lightweight Java application server
|
||||
- **Tech. description:** Java application JSP and servlet container
|
||||
@ -113,12 +137,13 @@ Components which are supplied as LXC overlay layers but don't run as standalone
|
||||
- **Used by:** MifosX, Motech, OpenDataKit, Sigmah
|
||||
- **Related skills:** JVM tuning, shell scripting
|
||||
|
||||
|
||||
## Components
|
||||
Components
|
||||
----------
|
||||
|
||||
Components which are supplied as LXC containers required by other applications but aren't exposed directly to the end user
|
||||
|
||||
### ActiveMQ
|
||||
ActiveMQ
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** Message broker middleware
|
||||
- **Tech. description:** Java-based message broker and messaging server
|
||||
@ -126,7 +151,8 @@ Components which are supplied as LXC containers required by other applications b
|
||||
- **Used by:** Motech
|
||||
- **Related skills:** JVM tuning, XML
|
||||
|
||||
### MariaDB
|
||||
MariaDB
|
||||
^^^^^^^
|
||||
|
||||
- **General description:** MySQL database server
|
||||
- **Tech. description:** MySQL-compatible relational database management system
|
||||
@ -134,15 +160,17 @@ Components which are supplied as LXC containers required by other applications b
|
||||
- **Used by:** MifosX, Ushahidi
|
||||
- **Related skills:** SQL (language overview)
|
||||
|
||||
### Postgres
|
||||
Postgres
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** PostgreSQL database server
|
||||
- **Tech. description:** PostgreSQL relational database management system
|
||||
- **Depends on:** -
|
||||
- **Used by:** CKAN, CrisisCleanup, CTS, GNU Health, KanBoard, Motech, Odoo, OpenDataKit, OpenDataKit Build, OpenMapKit, Pan.do/ra, Sahana Eden, SeedDMS, Sigmah
|
||||
- **Used by:** CKAN, CrisisCleanup, CTS, GNU Health, Kanboard, Motech, Odoo, OpenDataKit, OpenDataKit Build, OpenMapKit, Pan.do/ra, Sahana Eden, SeedDMS, Sigmah
|
||||
- **Related skills:** SQL (language overview)
|
||||
|
||||
### RabbitMQ
|
||||
RabbitMQ
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** Message broker middleware
|
||||
- **Tech. description:** Erlang-based message broker and messaging server
|
||||
@ -150,7 +178,8 @@ Components which are supplied as LXC containers required by other applications b
|
||||
- **Used by:** Pan.do/ra
|
||||
- **Related skills:** -
|
||||
|
||||
### Redis
|
||||
Redis
|
||||
^^^^^
|
||||
|
||||
- **General description:** Key-value NoSQL database server
|
||||
- **Tech. description:** In-memory NoSQL key-value data structure object store
|
||||
@ -158,7 +187,8 @@ Components which are supplied as LXC containers required by other applications b
|
||||
- **Used by:** CKAN
|
||||
- **Related skills:** -
|
||||
|
||||
### Solr
|
||||
Solr
|
||||
^^^^
|
||||
|
||||
- **General description:** Full-text search server
|
||||
- **Tech. description:** Apache Lucene-based full-text search and indexing platform
|
||||
@ -167,11 +197,13 @@ Components which are supplied as LXC containers required by other applications b
|
||||
- **Related skills:** shell scripting, JVM tuning
|
||||
|
||||
|
||||
## Applications
|
||||
Applications
|
||||
------------
|
||||
|
||||
Components which are supplied as LXC containers exposed directly to the end user
|
||||
|
||||
### CKAN
|
||||
CKAN
|
||||
^^^^
|
||||
|
||||
- **General description:** Data management and data store system
|
||||
- **Tech. description:** Flask-based data store application
|
||||
@ -179,7 +211,8 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** PostGIS, python frameworks and modules (flask, paster, setuptools), Solr core configuration
|
||||
|
||||
### CKAN DataPusher
|
||||
CKAN DataPusher
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
- **General description:** Data file parser for CKAN
|
||||
- **Tech. description:** Python-based data file parser service for CKAN
|
||||
@ -187,15 +220,17 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** CKAN
|
||||
- **Related skills:** MIME types identification, WSGI application development
|
||||
|
||||
### CrisisCleanup
|
||||
CrisisCleanup
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
- **General description:** Disaster response and relief coordination
|
||||
- **Tech. description:** Ruby-on-Rails-based application with Node.js-generated frontend assets
|
||||
- **Depends on:** LibXML, Node.js, Postgres, Ruby
|
||||
- **Tech. description:** Ruby-on-Rails-based application with Nodejs-generated frontend assets
|
||||
- **Depends on:** LibXML, Nodejs, Postgres, Ruby
|
||||
- **Used by:** User
|
||||
- **Related skills:** Node.js build systems (npm, yarn), Ruby build systems and frameworks (Bundle, Devise, Gems, Rails, Rake)
|
||||
- **Related skills:** Nodejs build systems (npm, yarn), Ruby build systems and frameworks (Bundle, Devise, Gems, Rails, Rake)
|
||||
|
||||
### CTS
|
||||
CTS
|
||||
^^^
|
||||
|
||||
- **General description:** Commodity tracking system
|
||||
- **Tech. description:** Django-based forms applicaton
|
||||
@ -203,7 +238,17 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** python frameworks (Django), WSGI application development
|
||||
|
||||
### FrontlineSMS
|
||||
Decidim
|
||||
^^^^^^^
|
||||
|
||||
- **General description:** Platform for citizen participation
|
||||
- **Tech. description:** Ruby on Rails-based forms and polls applicaton
|
||||
- **Depends on:** Nodejs, Postgres, Ruby
|
||||
- **Used by:** User
|
||||
- **Related skills:** Nodejs build systems (npm, yarn), Ruby build systems and frameworks (Bundle, Devise, Gems, Rails, Rake)
|
||||
|
||||
FrontlineSMS
|
||||
^^^^^^^^^^^^
|
||||
|
||||
- **General description:** Bulk text messaging
|
||||
- **Tech. description:** Spring-based application for modem device access
|
||||
@ -211,15 +256,17 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** Java build systems and frameworks (grails, Spring), Jetty, linux modem/tty handling, Xorg (X server)
|
||||
|
||||
### GNU Health
|
||||
GNU Health
|
||||
^^^^^^^^^^
|
||||
|
||||
- **General description:** Hospital information system (ERP)
|
||||
- **Tech. description:** Tryton-based form application RPC backend with Node.js-based frontend
|
||||
- **Depends on:** Node.js, Postgres, Python3
|
||||
- **Tech. description:** Tryton-based form application RPC backend with Nodejs-based frontend
|
||||
- **Depends on:** Nodejs, Postgres, Python3
|
||||
- **Used by:** User
|
||||
- **Related skills:** Node.js build systems (grunt, npm), python modules (setuptools, virtualenv)
|
||||
- **Related skills:** Nodejs build systems (grunt, npm), python modules (setuptools, virtualenv)
|
||||
|
||||
### KanBoard
|
||||
Kanboard
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:** Kanban project management
|
||||
- **Tech. description:** Symfony-based forms application
|
||||
@ -227,15 +274,17 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** PHP build systems and frameworks (Composer, Symfony)
|
||||
|
||||
### MifosX
|
||||
MifosX
|
||||
^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Spring-based forms application RPC backend with Node.js-packed frontend
|
||||
- **Tech. description:** Spring-based forms application RPC backend with Nodejs-packed frontend
|
||||
- **Depends on:** MariaDB, Tomcat
|
||||
- **Used by:** User
|
||||
- **Related skills:** Java frameworks (Spring), Node.js and JavaScript development (Angular, Bower, Grunt)
|
||||
- **Related skills:** Java frameworks (Spring), Nodejs and JavaScript development (Angular, Bower, Grunt)
|
||||
|
||||
### Motech
|
||||
Motech
|
||||
^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Apache Felix-based forms application
|
||||
@ -243,15 +292,17 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** Java frameworks (Apache Felix, Spring)
|
||||
|
||||
### Odoo
|
||||
Odoo
|
||||
^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Werkzeug-based forms application RPC backend with Node.js-based frontend
|
||||
- **Depends on:** Node.js, Postgres, Python 3
|
||||
- **Tech. description:** Werkzeug-based forms application RPC backend with Nodejs-based frontend
|
||||
- **Depends on:** Nodejs, Postgres, Python 3
|
||||
- **Used by:** User
|
||||
- **Related skills:** WSGI application development
|
||||
|
||||
### OpenDataKit
|
||||
OpenDataKit
|
||||
^^^^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Spring and OpenRosa-based data store application
|
||||
@ -259,23 +310,26 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** Java build systems and frameworks (Gradle, Spring)
|
||||
|
||||
### OpenDataKit Build
|
||||
OpenDataKit Build
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Ruby-based forms application with Node.js-based data converter
|
||||
- **Depends on:** Node.js, Ruby
|
||||
- **Tech. description:** Ruby-based forms application with Nodejs-based data converter
|
||||
- **Depends on:** Nodejs, Ruby
|
||||
- **Used by:** User
|
||||
- **Related skills:** Node.js build systems (npm), Ruby build systems and frameworks (Bundler, Rake)
|
||||
- **Related skills:** Nodejs build systems (npm), Ruby build systems and frameworks (Bundler, Rake)
|
||||
|
||||
### OpenMapKit
|
||||
OpenMapKit
|
||||
^^^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Node.js-based forms application with python2 dependencies
|
||||
- **Depends on:** Node.js, Postgres, Python 2
|
||||
- **Tech. description:** Nodejs-based forms application with python2 dependencies
|
||||
- **Depends on:** Nodejs, Postgres, Python 2
|
||||
- **Used by:** User
|
||||
- **Related skills:** JavaScript development, Node.js build systems (Yarn)
|
||||
- **Related skills:** JavaScript development, Nodejs build systems (Yarn)
|
||||
|
||||
### Pan.do/ra
|
||||
Pan.do/ra
|
||||
^^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Python-based media store with Javascript-based frontend
|
||||
@ -283,7 +337,8 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** ffmpeg, imagemagick, JavaScript development (oxjs)
|
||||
|
||||
### Sahana Eden
|
||||
Sahana Eden
|
||||
^^^^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Web2py-based forms application
|
||||
@ -291,7 +346,8 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** PostGIS, python frameworks and modules (requests, selenium, Web2py)
|
||||
|
||||
### SeedDMS
|
||||
SeedDMS
|
||||
^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** PHP-based data store application with Lucene-based fulltext index
|
||||
@ -299,7 +355,8 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** Apache Lucene, ghostscript, LibreOffice (unoconv), imagemagick, MIME types identification
|
||||
|
||||
### Sigmah
|
||||
Sigmah
|
||||
^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Apache Commons-based forms application
|
||||
@ -307,10 +364,11 @@ Components which are supplied as LXC containers exposed directly to the end user
|
||||
- **Used by:** User
|
||||
- **Related skills:** Java development (hibernate, log4j)
|
||||
|
||||
### Ushahidi
|
||||
Ushahidi
|
||||
^^^^^^^^
|
||||
|
||||
- **General description:**
|
||||
- **Tech. description:** Laravel-based RPC backend with Angular-based frontend
|
||||
- **Depends on:** MariaDB, PHP
|
||||
- **Used by:** User
|
||||
- **Related skills:** JavaScript build tools and frameworks (Angular, Babel, WebPck), PHP build tools and frameworks (composer, Laravel, phinx)
|
||||
- **Related skills:** JavaScript build tools and frameworks (Angular, Babel, WebPack), PHP build tools and frameworks (composer, Laravel, phinx)
|
||||
37
doc/conf.py
37
doc/conf.py
@ -1,21 +1,4 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
#
|
||||
# Configuration file for the Sphinx documentation builder.
|
||||
#
|
||||
# This file does only contain a selection of the most common options. For a
|
||||
# full list see the documentation:
|
||||
# http://www.sphinx-doc.org/en/master/config
|
||||
|
||||
# -- Path setup --------------------------------------------------------------
|
||||
|
||||
# If extensions (or modules to document with autodoc) are in another directory,
|
||||
# add these directories to sys.path here. If the directory is relative to the
|
||||
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
||||
#
|
||||
# import os
|
||||
# import sys
|
||||
# sys.path.insert(0, os.path.abspath('.'))
|
||||
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
|
||||
@ -31,27 +14,9 @@ release = ''
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
|
||||
# If your documentation needs a minimal Sphinx version, state it here.
|
||||
#
|
||||
# needs_sphinx = '1.0'
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = ['recommonmark', 'sphinx_markdown_tables']
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
|
||||
# The suffix(es) of source filenames.
|
||||
# You can specify multiple suffix as a list of string:
|
||||
#
|
||||
# source_suffix = ['.rst', '.md']
|
||||
source_suffix = {
|
||||
'.rst': 'restructuredtext',
|
||||
'.md': 'markdown'
|
||||
}
|
||||
|
||||
# The master toctree document.
|
||||
master_doc = 'index'
|
||||
|
||||
@ -133,7 +98,7 @@ latex_elements = {
|
||||
# author, documentclass [howto, manual, or own class]).
|
||||
latex_documents = [
|
||||
(master_doc, 'SpotterVM.tex', 'SpotterVM Documentation',
|
||||
'Disassembler', 'manual'),
|
||||
'Spotter', 'manual'),
|
||||
]
|
||||
|
||||
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
Information about existing containers
|
||||
=====================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
list
|
||||
tech-overview
|
||||
map-services
|
||||
@ -1,57 +0,0 @@
|
||||
# List of existing containers
|
||||
|
||||
## List of basic and runtime layers
|
||||
|
||||
| Layer | Container |
|
||||
|-------------------------|---------------------|
|
||||
| Alpine 3.8 | alpine3.8 |
|
||||
| Alpine 3.8 - PHP 5.6 | alpine3.8-php5.6 |
|
||||
| Alpine 3.9 - Ruby 2.4 | alpine3.8-ruby2.4 |
|
||||
| Alpine 3.9 | alpine3.9 |
|
||||
| Alpine 3.9 - Java 8 | alpine3.9-java8 |
|
||||
| Alpine 3.9 - PHP 7.2 | alpine3.9-php7.2 |
|
||||
| Alpine 3.9 - Python 2.7 | alpine3.9-python2.7 |
|
||||
| Alpine 3.9 - Python 3.6 | alpine3.9-python3.6 |
|
||||
| Alpine 3.9 - Ruby 2.4 | alpine3.9-ruby2.4 |
|
||||
| Alpine 3.9 - Ruby 2.6 | alpine3.9-ruby2.6 |
|
||||
| Alpine 3.9 - Tomcat 7 | alpine3.9-tomcat7 |
|
||||
| Alpine 3.9 - Tomcat 8.5 | alpine3.9-tomcat8.5 |
|
||||
|
||||
## List of service containers
|
||||
|
||||
| Service | Container | UID/GID | Internal Port |
|
||||
|-----------------|-----------------|---------|------------------|
|
||||
| ActiveMQ | activemq | 61616 | 61616 (ActiveMQ) |
|
||||
| MariaDB | mariadb | 3306 | 3306 (MySQL) |
|
||||
| Postgres | postgres | 5432 | 5432 (Postgres) |
|
||||
| PostGIS | postgis | 5432 | 5432 (Postgres) |
|
||||
| RabbitMQ | rabbitmq | 5672 | 5672 (AMQP) |
|
||||
| Redis | redis | 6379 | 6379 (Redis) |
|
||||
| Solr 6 | solr6 | 8983 | 8983 (HTTP) |
|
||||
|
||||
## List of application containers
|
||||
|
||||
All application containers have the application user UID/GID 8080 and listen on internal port 8080 (HTTP)
|
||||
|
||||
| Application | Container | Host |
|
||||
|----------------|-------------------|-------------|
|
||||
| CKAN | ckan | ckan |
|
||||
| Crisis Cleanup | crisiscleanup | cc |
|
||||
| CTS | cts | cts |
|
||||
| EcoGIS | ecogis | ecogis |
|
||||
| FrontlineSMS | frontlinesms | sms |
|
||||
| GNU Health | gnuhealth | gh |
|
||||
| KanBoard | kanboard | kb |
|
||||
| Mifos X | mifosx | mifosx |
|
||||
| Motech | motech | motech |
|
||||
| ODK Aggregate | opendatakit | odk |
|
||||
| ODK Build | opendatakit-build | odkbuild |
|
||||
| Odoo | odoo | odoo |
|
||||
| OpenMapKit | openmapkit | omk |
|
||||
| Pan.do/ra | pandora | pandora |
|
||||
| Sahana | sahana | sahana |
|
||||
| Sahana - Demo | sahana-demo | sahana-demo |
|
||||
| SAMBRO | sambro | sambro |
|
||||
| SeedDMS | seeddms | dms |
|
||||
| Sigmah | sigmah | sigmah |
|
||||
| Ushahidi | ushahidi | ush |
|
||||
@ -1,27 +0,0 @@
|
||||
# Map services used in applications
|
||||
|
||||
| Application | Data sources | Map viewer | Configurable | Notes |
|
||||
|---------------|--------------|-------------|--------------|-------|
|
||||
| CKAN | OSM ([Stamen](http://maps.stamen.com)) | Leaflet | No | [1] |
|
||||
| CrisisCleanup | Google Maps | Google Maps | No | |
|
||||
| CTS | OSM, [ArcGIS](http://server.arcgisonline.com/arcgis/rest/services) | Leaflet | No | |
|
||||
| EcoGIS | ? | FreeGIS + OpenLayers 2 | Probably | [2] |
|
||||
| Odoo | Google Maps | Google Maps | No | [3] |
|
||||
| OpenMapKit | OSM | N/A | Yes | [4] |
|
||||
| Pan.do/ra | Google Maps | Google Maps + OxMap | No | |
|
||||
| Sahana Eden | OSM ([HOT](https://www.hotosm.org/)) | OpenLayers 2 | Yes, very | [5] |
|
||||
| Ushahidi | OSM ([Mapbox](https://www.mapbox.com/about/maps/), [HOT](https://www.hotosm.org/)) | Leaflet | No | |
|
||||
| --- | | | | |
|
||||
| WebODM | OSM | Leaflet | | |
|
||||
| Crismapp | OSM | Leaflet | | |
|
||||
| ThinkHazard! | OSM | Mapbox | | |
|
||||
| Openforis | Google Earth | | No | |
|
||||
| Tendenci | Google Maps | Google Maps | No | |
|
||||
| GeoNode | OSM | ? OpenLayers 3 | | |
|
||||
|
||||
|
||||
1. Used by CKAN extensions *reclineview*, *spatial* and *geoview*.
|
||||
2. Untested as the EcoGIS source code is not fully open. Looks like the data sources are configurable, but the full documentation is only in italian.
|
||||
3. Used by Odoo *Google Maps* module to display company/partner address on map.
|
||||
4. Map is used by OMK and ODK Android clients. OMK Server only offers the API. See [area of interest depoyment](http://posm.io/docs/omk/walkthrough/) on POSM wiki.
|
||||
5. Sahana Eden supports multitude of connectors and protocols to process map and feature data. ArcGIS REST, Bing maps, GeoJSON, GeoRSS, Google Maps, GPX, KML, MGRS, OSM, OWM, Shapefile, TMS, WFS, WMS and XYZ.
|
||||
@ -6,4 +6,4 @@ Welcome to SpotterVM documentation!
|
||||
:maxdepth: 3
|
||||
|
||||
toolchain/index
|
||||
existing/index
|
||||
applications/index
|
||||
|
||||
@ -1,49 +0,0 @@
|
||||
# Alpine build and packaging
|
||||
|
||||
Alpine build system is used for all custom non-LXC packages, such as Acme.sh Let's Encrypt ACME client and VMMgr virtual machine manager.
|
||||
|
||||
## Alpine wiki references
|
||||
|
||||
The usage of Abuild, APK package manager and syntax of `APKBUILD` files is best described on Alpine wiki.
|
||||
|
||||
- [Abuild and Helpers](https://wiki.alpinelinux.org/wiki/Abuild_and_Helpers)
|
||||
- [Creating an Alpine package](https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package)
|
||||
- [APKBUILD Reference](https://wiki.alpinelinux.org/wiki/APKBUILD_Reference)
|
||||
- [APKBUILD Examples](https://wiki.alpinelinux.org/wiki/APKBUILD_examples)
|
||||
|
||||
## Abuild in a nutshell
|
||||
|
||||
Building with abuild requires `alpine-sdk` package installed, `/etc/abuild.conf` configured and an RSA private key created in `/srv/repo.spotter.cz.rsa` and subsequently registered by `abuild-keygen` command. All these are taken care of in `install-toolchain.sh` script as part of [Build environment installation](vm-creation).
|
||||
|
||||
Abuild toolchain is intended to be used in automated builds, therefore it requires some dependencies normally not found in other packaging systems. Abuild expects that `APKBUILD` files are a part of git repository and tries to read current commit hash. Then it tries to automatically download, build (compile), strip binaries, find out dependencies, and generally perform a lot of tasks normally useful when you are compiling from sources. Finally it packages the result to one or more subpackages, according to the build recipe. For purposes of LXC packaging, this is mostly useless, which is the reason why we have a [custom package manager](pkgmgr). It is however perfectly suitable for packages installed directly on the basic VM.
|
||||
|
||||
## APKFILE and build example
|
||||
|
||||
Following is an `APKFILE` example with some commonly used options to skip or bypass the default features of Abuild in case you simply need to package a bunch of existing files without any compilation or other build tasks.
|
||||
|
||||
```
|
||||
# Contributor: Disassembler <disassembler@dasm.cz>
|
||||
# Maintainer: Disassembler <disassembler@dasm.cz>
|
||||
pkgname=somepackage
|
||||
pkgver=0.0.1
|
||||
pkgrel=0
|
||||
pkgdesc="Some description"
|
||||
url="https://spotter.vm/"
|
||||
arch="noarch"
|
||||
license="GPL"
|
||||
depends="python3"
|
||||
options="!check !strip"
|
||||
|
||||
build() {
|
||||
return 0
|
||||
}
|
||||
|
||||
package() {
|
||||
mkdir -p ${pkgdir}
|
||||
cp -rp mydir ${pkgdir}
|
||||
}
|
||||
```
|
||||
|
||||
The directive `options="!check !strip"` requests Abuild not to run post-build checks and not to strip binaries. The `build()` function is mandated by the Abuild documentation and simply returns an exit code without doing anything. Finally in `package()` function, the desired existing files are copied to `${pkgdir}` (which is a variable automatically set by Abuild) and packaged.
|
||||
|
||||
Such `APKFILE` recipe is then executed using `abuild` command. Abuild normally uses `fakeroot` to isolate the build environment and discourages using root user for packaging, however our build instance is highly specialized for this purpose, so we package as root anyway. Any user (including root) needs to be a member of `abuild` group in order to perform the task. For our root user, this is again handled in `install-toolchain.sh`. If you do the packaging as root, you need to run `abuild -F` as seen in `build-all.sh`.
|
||||
53
doc/toolchain/abuild.rst
Normal file
53
doc/toolchain/abuild.rst
Normal file
@ -0,0 +1,53 @@
|
||||
Alpine build and packaging
|
||||
==========================
|
||||
|
||||
Alpine build system (abuild) is used for native Alpine linux packages, such as LXC binaries, SPOC and VMMgr virtual machine manager.
|
||||
|
||||
Alpine wiki references
|
||||
----------------------
|
||||
|
||||
Usage of abuild, APK (*Alpine Package Keeper*) manager and syntax of ``APKBUILD`` files is best described on Alpine wiki.
|
||||
|
||||
- `Abuild and Helpers <https://wiki.alpinelinux.org/wiki/Abuild_and_Helpers>`_
|
||||
- `Creating an Alpine package <https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package>`_
|
||||
- `APKBUILD Reference <https://wiki.alpinelinux.org/wiki/APKBUILD_Reference>`_
|
||||
- `APKBUILD Examples <https://wiki.alpinelinux.org/wiki/APKBUILD_examples>`_
|
||||
|
||||
Abuild in a nutshell
|
||||
--------------------
|
||||
|
||||
Building with abuild requires ``alpine-sdk`` package installed, ``/etc/abuild.conf`` configured and an RSA private key created in ``/root/repo.spotter.cz.rsa`` and subsequently registered by ``abuild-keygen`` command. All these are taken care of in ``install-toolchain.sh`` script as part of `Build environment installation <vm-creation>`_.
|
||||
|
||||
Abuild toolchain is intended to be used in automated builds, therefore it requires some dependencies normally not found in other packaging systems. Abuild expects that ``APKBUILD`` files are a part of git repository and tries to read current commit hash. Then it tries to automatically download, build (compile), strip symbols binaries, find out dependencies, and generally perform a lot of tasks normally useful when you are compiling binaries from sources. Finally it packages the result to one or more subpackages, according to the build recipe. For purposes of SPOC containers packaging, this is mostly useless, which is the reason why we have a `custom package manager <spoc>`_. It is however perfectly suitable for packages installed directly on the basic VM.
|
||||
|
||||
APKFILE and build example
|
||||
-------------------------
|
||||
|
||||
Following is an ``APKFILE`` example with some commonly used options to skip or bypass the default features of abuild in case you simply need to package a bunch of existing files without any compilation or other build tasks.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Contributor: Disassembler <disassembler@dasm.cz>
|
||||
# Maintainer: Disassembler <disassembler@dasm.cz>
|
||||
pkgname=somepackage
|
||||
pkgver=0.0.1
|
||||
pkgrel=0
|
||||
pkgdesc="Some description"
|
||||
url="https://spotter.vm/"
|
||||
arch="noarch"
|
||||
license="GPL"
|
||||
depends="python3"
|
||||
options="!check !strip"
|
||||
|
||||
build() {
|
||||
return 0
|
||||
}
|
||||
|
||||
package() {
|
||||
mkdir -p ${pkgdir}
|
||||
cp -rp mydir ${pkgdir}
|
||||
}
|
||||
|
||||
The directive ``options="!check !strip"`` requests abuild not to run post-build checks and not to strip symbols from binaries. The ``build()`` function is mandated by the abuild documentation and simply returns success exit code without doing anything. Finally in ``package()`` function, the desired existing files are copied to ``${pkgdir}`` (which is a variable automatically set by abuild) and packaged.
|
||||
|
||||
Such ``APKFILE`` recipe is then executed using ``abuild`` command. Abuild normally uses ``fakeroot`` to isolate the build environment and discourages using root user for packaging, however our build instance is highly specialized for this purpose, so we package as root anyway. Any user (including root) needs to be a member of ``abuild`` group in order to perform the task. For our root user, this is again handled in ``install-toolchain.sh``. If you do the packaging as root, you need to run ``abuild -F`` as seen in ``build-all.sh``.
|
||||
@ -1,13 +1,15 @@
|
||||
VM building and packaging
|
||||
=========================
|
||||
Toolchain
|
||||
=========
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
vm-creation
|
||||
virtual-machine-creation
|
||||
virtual-machine-internals
|
||||
abuild
|
||||
lxc-overview
|
||||
lxcbuild
|
||||
lxc-pack
|
||||
pkgmgr
|
||||
vmmgr-hooks
|
||||
spoc-overview
|
||||
spoc-architecture
|
||||
spoc-builder
|
||||
spoc-user
|
||||
vmmgr-overview
|
||||
vmmgr-internals
|
||||
|
||||
@ -1,183 +0,0 @@
|
||||
# Building LXC containers
|
||||
|
||||
## Overview
|
||||
|
||||
`lxcbuild` utility creates a LXC container based on its build recipe and build context path given in command line parameter. If a filename is given, the build recipe is loaded from the file and the directory in which the file resides is taken as build context, ie. all relative paths are resolved from it. In case a directory path is passed as parameter, the directory is then used as build context and a file called `lxcfile` from the given directory is used as build recipe.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
lxcbuild <buildpath>
|
||||
where the buildpath can be either specific lxcfile or a directory containing one
|
||||
```
|
||||
|
||||
## Directives used in lxcfile
|
||||
|
||||
The *lxcfile* syntax is designed to resemble *Dockerfile* syntax in order to ease the potential transition. Since LXC operates on much lower level of abstraction than Docker, some principles are applied more explicitly and verbosely. Major difference between Docker and *lxcbuild* is that every directive in *Dockerfile* creates a new filesystem layer whereas layers in *lxcbuild* are managed manually.
|
||||
|
||||
### IMAGE
|
||||
|
||||
- **Usage:** `IMAGE <name>`
|
||||
- **Description:** Sets container name. Every *lxcfile* needs to have one, otherwise no LXC config can be written and no `RUN` scripts can be run. LXC configuration file is written to path `/var/lib/lxc/<name>/config`
|
||||
- **Docker equivalent:** `-t` in `docker build` command line parameters
|
||||
- **Populates LXC field:** `lxc.uts.name`
|
||||
|
||||
### LAYER
|
||||
|
||||
- **Usage:** `LAYER <path>`
|
||||
- **Description:** Includes OverlayFS layer. Unlike *Dockerfile*'s `FROM`, in *lxcfile* all layers need to be defined. The `LAYER` directives are given from the lowermost to the uppermost layer, where the lowermost is usually the basic operating system and the uppermost is the working layer in which all subsequent `RUN` commands and other action will take place.
|
||||
- **Docker equivalent:** `FROM`
|
||||
- **Populates LXC field:** `lxc.rootfs.path`
|
||||
|
||||
### RUN
|
||||
|
||||
- **Usage:**
|
||||
|
||||
```bash
|
||||
RUN <label>
|
||||
<commands>
|
||||
<label>
|
||||
```
|
||||
|
||||
- **Description:** Executes a shell script in the currently built container. The `<label>` is an arbitrary user defined string which needs to be given as the first parameter and repeated at the end of the script block. The shell script between the labels is passed as-is, including comments and empty lines, to a POSIX shell with `-e` and `-v` parameters set. Basically, following *lxcfile* entry:
|
||||
|
||||
```bash
|
||||
RUN EOF
|
||||
# Comment
|
||||
command1
|
||||
command2
|
||||
EOF
|
||||
```
|
||||
|
||||
translates to the following script:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
set -ev
|
||||
|
||||
# Comment
|
||||
command1
|
||||
command2
|
||||
```
|
||||
|
||||
The command chaining via `&&` which is required in *Dockerfile* is optional in *lxcbuild*.
|
||||
|
||||
- **Docker equivalent:** `RUN`
|
||||
- **Populates LXC field:** None
|
||||
|
||||
### COPY
|
||||
|
||||
- **Usage:** `COPY <source> [destination]`
|
||||
- **Description:** Recursively copies `<source>` files into `<destination>`. Source path is relative to the build context directory, destination path is relative to the container root directory. The files are copied as `root:root`. Permissions can be changed by subsequent `RUN` command.
|
||||
|
||||
The `<source>` can be given as http:// or https:// URL in which case gzip, bzip2 or xz tar archive is expected to be downloaded and unpacked into the `<destination>`. This is commonly used for creating a basic root filesystem of the container in similar fashion like with Docker's `FROM scratch`.
|
||||
|
||||
- **Docker equivalent:** `COPY` or `ADD`
|
||||
- **Populates LXC field:** None
|
||||
|
||||
### USER
|
||||
|
||||
- **Usage:** `USER <uid> <gid>`
|
||||
- **Description:** Sets UID/GID of the container init process to `<uid>` and `<gid>`. The default UID/GID is `0:0 (root:root)`.
|
||||
- **Docker equivalent:** `USER`
|
||||
- **Populates LXC field:** `lxc.init.uid` and `lxc.init.gid`
|
||||
|
||||
### CMD
|
||||
|
||||
- **Usage:** `CMD <command> [parameters...]`
|
||||
- **Description:** Sets the init process of the container. This is the process which is automatically started after the container is launched. The default command is `/bin/true` which immediately terminates with return code 0.
|
||||
- **Docker equivalent:** `CMD`
|
||||
- **Populates LXC field:** `lxc.init.cmd`
|
||||
|
||||
### ENV
|
||||
|
||||
- **Usage:** `ENV <variable> <value>`
|
||||
- **Description:** Populates environment variable `<variable>` with `<value>` which is then passed to the init process when the container is launched.
|
||||
- **Docker equivalent:** `ENV`
|
||||
- **Populates LXC field:** `lxc.environment`
|
||||
|
||||
### WORKDIR
|
||||
|
||||
- **Usage:** `WORKDIR <dirname>`
|
||||
- **Description:** Sets working directory of the container init process to `<dirname>`. The default working directory is the container's root directory.
|
||||
- **Docker equivalent:** `WORKDIR`
|
||||
- **Populates LXC field:** `lxc.init.uid` and `lxc.init.gid`
|
||||
|
||||
### HALT
|
||||
|
||||
- **Usage:** `HALT <signal>`
|
||||
- **Description:** Sets container stop signal to `<signal>`. The default signal is SIGINT.
|
||||
- **Docker equivalent:** `--signal` in `docker kill` command line parameters
|
||||
- **Populates LXC field:** `lxc.signal.halt`
|
||||
|
||||
## LXC config
|
||||
|
||||
Although *lxcfile* populates some LXC config fields, there are lot of defaults with remain unchanged. The template file to which *lxcbuild* fills in the values looks as follows:
|
||||
|
||||
```bash
|
||||
# Image name
|
||||
lxc.uts.name = {name}
|
||||
|
||||
# Network
|
||||
lxc.net.0.type = veth
|
||||
lxc.net.0.link = lxcbr0
|
||||
lxc.net.0.flags = up
|
||||
|
||||
# Volumes
|
||||
lxc.rootfs.path = {rootfs}
|
||||
|
||||
# Mounts
|
||||
lxc.mount.entry = shm dev/shm tmpfs rw,nodev,noexec,nosuid,relatime,mode=1777,create=dir 0 0
|
||||
lxc.mount.entry = /etc/hosts etc/hosts none bind,create=file 0 0
|
||||
lxc.mount.entry = /etc/resolv.conf etc/resolv.conf none bind,create=file 0 0
|
||||
{mounts}
|
||||
|
||||
# Init
|
||||
lxc.init.cmd = {cmd}
|
||||
lxc.init.uid = {uid}
|
||||
lxc.init.gid = {gid}
|
||||
lxc.init.cwd = {cwd}
|
||||
|
||||
# Environment
|
||||
lxc.environment = PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
{env}
|
||||
|
||||
# Halt
|
||||
lxc.signal.halt = {halt}
|
||||
|
||||
# Log
|
||||
lxc.console.size = 1MB
|
||||
lxc.console.logfile = /var/log/lxc/{name}.log
|
||||
|
||||
# Other
|
||||
lxc.arch = x86_64
|
||||
lxc.cap.drop = sys_admin
|
||||
lxc.hook.pre-start = /usr/bin/vmmgr prepare-container
|
||||
lxc.hook.start-host = /usr/bin/vmmgr register-container
|
||||
lxc.hook.post-stop = /usr/bin/vmmgr unregister-container
|
||||
lxc.include = /usr/share/lxc/config/common.conf
|
||||
```
|
||||
|
||||
For explanation of hooks and overall container integration and behavior, refer to [VMMgr hooks](vmmgr-hooks) page.
|
||||
|
||||
## Example lxcfile
|
||||
|
||||
Following is an example of *lxcfile* for *Redis*:
|
||||
|
||||
```bash
|
||||
IMAGE redis
|
||||
LAYER shared/alpine
|
||||
LAYER redis/redis
|
||||
|
||||
RUN EOF
|
||||
# Create OS user (which will be picked up later by apk add)
|
||||
addgroup -S -g 6379 redis
|
||||
adduser -S -u 6379 -h /var/lib/redis -s /bin/false -g redis -G redis redis
|
||||
|
||||
# Install Redis
|
||||
apk --no-cache add redis
|
||||
EOF
|
||||
|
||||
USER 6379 6379
|
||||
CMD redis-server /etc/redis.conf
|
||||
```
|
||||
@ -1,71 +0,0 @@
|
||||
# LXC containers overview
|
||||
|
||||
All user-installable applications run in LXC containers. A *container* is defined by a configuration file with following settings:
|
||||
|
||||
- Network type and interface configuration
|
||||
- OverlayFS storage layers
|
||||
- Mountpoints to store / load persistent data
|
||||
- Functional user and binary to be executed on startup
|
||||
- Environment variables propagated to the container namespace
|
||||
- Signal used to stop the container
|
||||
- TTY / console logging
|
||||
- Syscall capability restrictions
|
||||
- Event hooks
|
||||
|
||||
The container must have at least one storage layer defined. The term *layer* is used because the storage is handled by OverlayFS filesystem consisting of groups of files overlaid over each other. This allows to have a layer with the basic operating system, another layer with python runtime, and final layer with the application using both. Therefore it's not necessary to duplicate the functionality in every container and waste disk space.
|
||||
|
||||
Each layer is then packaged to a separate installable package and given as dependencies to the packages which require them. Packages with final application layer also contain the container configuration and installation scripts to set up the application and interface it with other components installed on host and in other containers.
|
||||
|
||||
## Why LXC
|
||||
|
||||
There are several container runtimes, with *Docker* being probably the most popular nowadays. There are several reasons why LXC was eventually selected instead.
|
||||
|
||||
First and foremost, Docker contains a huge set of tools for use with various orchestrators and large-scale applications. The premise of Docker is to run multiple instances of application containers where the individual instances are configured on runtime via command line parameters. Docker daemon and its shim processes contain a lot of abstraction, effectively obstructing the visibility on what is actually going on under the hood. LXC, on the other hand, keeps thing close to the bare minimum and transparently uses container techniques, syscalls and namespaces exposed by the linux kernel. The containers in LXC are fully defined via configuration files and don't require any additional configuration on runtime. This can arguably be achieved even on Docker via *docker-composer*, but that adds yet another layer of abstraction and generally is not suitable for scenarios where the container images need to be added or removed on the fly.
|
||||
|
||||
Docker is written in Go language, which is designed to create runtime-safe statically linked executables. With the shear amount of Docker capabilities, this unfortunately means that the whole Docker infrastructure occupies roughly 200 MB on the VM hard drive. The basic virtual machine image is designed to be as small as possible, so having a 200 MB large container host on an operating system which alone occupies roughly 40 MB does not seem ideal. LXC runtime written in C/C++ on the other hand occupies roughly 4 MB and doesn't need any other dependencies besides *cgroupfs* which, for performance reasons, is good to have installed anyway.
|
||||
|
||||
Due to the Docker's approach, storage overlay layers cannot be easily managed by the container builder and instead depend on the amount and order of directives in *Dockerfile* recipe file. This often leads to duplication of layers just because they are in slightly different order than another container has. So if one container has order of layers *system* -> *python* -> *java* and another has *system* -> *java* -> *nodejs*, only the *system will be shared but the *java* will be duplicated. This of course makes sense if reordering the layers makes the final content inconsistent, however this is not the case with Alpine linux (there is one specific case where it is a problem, but it can be circumvented), so with LXC, we have a full control on what will be in a single layer and in which order will the layers be overlaid.
|
||||
|
||||
Finally, Docker maintainers explicitly refuse to implement a possibility to isolate the docker daemon to private Docker repositories (registries) in the community edition of Docker. It is possible to have some custom and even private repositories, but it is not possible to deactivate the default public *Dockerhub*.
|
||||
|
||||
The downsides of using LXC is that its usage requires a bit more knowledge about how the linux containers actually work, and that most 3rd party applications are distributed using `Dockerfile`, which requires rewriting into LXC, however this is simplified by the [`lxcbuild`](lxcbuild) tool, which aims to automatize LXC container building using *Dockerfile*-like syntax.
|
||||
|
||||
## Container interfaces
|
||||
|
||||
Due to the fact that LXC container has all environment variables, mounts, used layers, init and everything it needs for starting, running and stopping in its `config` configuration file, you are completely free to build the container in any way possible. The only requirement imposed by the host (virtual machine) infrastructure is that in case of containers with user-accessible web interfaces via HTTP(S) which needs to be proxied via nginx HTTP server, the application needs to be reachable via plain HTTP on port 8080/TCP.
|
||||
|
||||
The container itself is normally handled as a service via hosts OpenRC init scripts, calling `lxc-start` and `lxc-stop`. Should the application in the container be user-accessible, it needs to register itself in host's nginx HTTP proxy server via hooks described in [VMMgr hooks](vmmgr-hooks). Full example of an application container init script is as follows:
|
||||
|
||||
```bash
|
||||
#!/sbin/openrc-run
|
||||
|
||||
description="CKAN container"
|
||||
|
||||
depend() {
|
||||
need ckan-datapusher postgres redis solr
|
||||
}
|
||||
|
||||
start() {
|
||||
lxc-start ckan
|
||||
}
|
||||
|
||||
start_post() {
|
||||
vmmgr register-proxy ckan
|
||||
}
|
||||
|
||||
stop_pre() {
|
||||
vmmgr unregister-proxy ckan
|
||||
}
|
||||
|
||||
stop() {
|
||||
lxc-stop ckan
|
||||
}
|
||||
```
|
||||
|
||||
See [`openrc-run(8)`](http://manpages.org/openrc-run/8) manual page for reference.
|
||||
|
||||
## Recommended tools and practices
|
||||
|
||||
If the application itself doesn't support connection via plain HTTP (e.g. it is a CGI/WSGI application), the container needs to contain also a web server which will proxy the connection. Recommended web server for this purpose is *nginx* HTTP server, which is lightweight and can proxy all commonly used gateway interfaces. Bear in mind that in some cases, the application needs to able to infer its own HTTP host, so some header tuning for both HTTP and CGI/WSGI protocols might be in order.
|
||||
|
||||
In case there are more components or services running within the same container (e.g. nginx HTTP server and PHP-FPM), it is advised to have them spawned and supervised using some lightweight init daemon. Recommended init system for LXC is *s6*, however if you are familiar with *daemontools* or *runit*, feel free to use them as well. In the worst case you can use *OpenRC*. Systems like *Upstart*, *Systemd* or *SysV init* are not recommended for their complexity or inability to properly supervise the spawned processes.
|
||||
@ -1,78 +0,0 @@
|
||||
# Packaging LXC containers
|
||||
|
||||
## Overview
|
||||
|
||||
The `lxc-pack` utility creates a `.tar.xz` archives based on package metadata and manages the `packages.json` repository metadata file. If a filename is passed as command line parameter to `lxc-pack`, the metadata are loaded from the file. In case a directory path is given, the metadata are loaded from a file called `pkg` from the directory. All metadata files are in JSON format.
|
||||
|
||||
The product of *lxcbuild* command described in LXC building documentation can be used in its entirety, ie. both filesystem layer and configuration, or only as dependency, in which case the container configuration is omitted and only the filesystem layer is used. Apart from that, the package can contain installation, upgrade and uninstallation script and data, all of which are optional. Accepted names are
|
||||
|
||||
- `install.sh` file and `install` directory for post-install scripts.
|
||||
- `upgrade.sh` file and `upgrade` directory for post-upgrade scripts.
|
||||
- `uninstall.sh` file and `uninstall` directory for post-uninstall scripts.
|
||||
|
||||
`lxc-pack` reads the metadata file, creates a tarball with the contents of a given directory under `/var/lib/lxc`, adds the install/upgrade/uninstall scripts and compresses the tarball into `.tar.xz` archive. Then it calculated size of the package and SHA512 hash and adds this information to the rest of the metadata, which are then stored as JSON dictionary in the repository-wide metadata file `packages`. Finally, it creates another SHA512 hash of the `packages` file and signs it using ECDSA key to ensure the integrity and tamperproofness of the repository data. The signature is stored in `packages.sig` file. Public key for the signature verification is already pre-imported on the LXC hosts as part of the basic VM installation. For more details on package manager, see the Package Manager documentation.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
lxc-pack <buildpath>
|
||||
where the buildpath can be either specific meta file or a directory containing one
|
||||
```
|
||||
|
||||
## Keys used in meta file
|
||||
|
||||
The `meta` file is in JSON format. All values are strings except for `depends` which is an array of strings and `size` which is an integer.
|
||||
|
||||
### title
|
||||
|
||||
- **Usage:** `"title": "<title>"`
|
||||
- **Description:** Sets human readable package name. Also helps to distinguish if the package is user-installable or if it is a dependency / component to another application. All packages which have title set, will show up in a list of user-installable packages in VMMgr web GUI.
|
||||
- **Mandatory:** Only for full user-installable container packages.
|
||||
|
||||
### desc-xx
|
||||
|
||||
- **Usage:** `"desc-<lang>": "<description>"`
|
||||
- **Description:** Sets human readable long description of the package. Language code `lang` is ISO 639-1 two-letter code.
|
||||
- **Mandatory:** Only for full user-installable container packages.
|
||||
|
||||
### lxcpath
|
||||
|
||||
- **Usage:** `"lxcpath": "<directory>[/subdirectory]"`
|
||||
- **Description:** Sets the source path for `lxc-pack` and subsequently also for VMMgr. The `directory` is a relative path under `/var/lib/lxc`. In case only the directory is given, `lxc-pack` takes all subdirectories and files in the directory, usually resulting in a full container package (both filesystem layer and configuration). If a `/subdirectory` is given, then only the subdirectory is packaged, resulting in filesystem layer-only package, usually used as a shared dependency for other containers.
|
||||
- **Mandatory:** Yes.
|
||||
|
||||
### version
|
||||
|
||||
- **Usage:** `"version": "<version>"`
|
||||
- **Description:** Sets the package version. This should correspond to the actual version of the packaged product.
|
||||
- **Mandatory:** Yes.
|
||||
|
||||
### release
|
||||
|
||||
- **Usage:** `"release": "<release>"`
|
||||
- **Description:** Sets the package release version. Used when the same basic version of the packaged product needs to be repacked with updated base layers or install/upgrade/uninstall scripts.
|
||||
- **Mandatory:** Yes.
|
||||
|
||||
### license
|
||||
|
||||
- **Usage:** `"license": "<license>"`
|
||||
- **Description:** Sets the license of the packaged product.
|
||||
- **Mandatory:** Yes.
|
||||
|
||||
### depends
|
||||
|
||||
- **Usage:** `"depends: ["<dependency1>", "<dependency2>", ...]`
|
||||
- **Description:** Sets the package dependencies which needs to be installed before this package.
|
||||
- **Mandatory:** Yes (can be empty for the basic OS filesystem layer).
|
||||
|
||||
### size
|
||||
|
||||
- **Usage:** `"size": "<bytes>"`
|
||||
- **Description:** Sets the package archive size.
|
||||
- **Mandatory:** Populated automatically by `lxc-pack`.
|
||||
|
||||
### sha512
|
||||
|
||||
- **Usage:** `"sha512": "<hash>"`
|
||||
- **Description:** Sets the package archive SHA512 hash.
|
||||
- **Mandatory:** Populated automatically by `lxc-pack`.
|
||||
@ -1,73 +0,0 @@
|
||||
# Package manager
|
||||
|
||||
## Why custom package manager
|
||||
|
||||
Why use custom package management instead of the native APK package manager?
|
||||
|
||||
Native packaging toolchain [`abuild`](abuild) is designed for automated bulk package building. It doesn't support building packages from pre-existing directories without some considerable customizations and requires that full build takes place as part of the packaging process. That includes steps like binary stripping, symlink resolution and dependency tracing. It also requires to be run under non-root user inside `fakeroot` which is problematic when LXC containers should be packaged. Most of the limitations can be worked around (run as root using `-F`, spoof build process by bind-mounting existing directory to packaging directory, skip dependency tracing using `options="!tracedeps"` in `APKFILE` and omit majority of the build process by running only `build package prepare_metafiles create_apks index clean` abuild actions), however there is no real benefit in (ab)using the native tools this way.
|
||||
|
||||
Furthermore, when `apk` package manager installs a package, it first unpacks it, then runs post-install script and once all packages are installed, only then it modifies the permissions and ownership of the files to the original values contained in the package. This means that it's not possible to run container setup as part of post-install script as most applications require the permissions to be already correct. Every single file including its ownership and permissions along with a hash is recorded in `/lib/apk/db/installed`, which only unnecessarily bloats the database of locally installed packages (e.g. the basic python 3 layer contains ~6500 files).
|
||||
|
||||
With custom package manager, the whole download, unpacking and installation process can be observed directly, keeping the VMMgr web GUI user informed about the currently ongoing step, as opposed to a mere download percentage offered by the bare `apk`. Finally, the APK packages are only gzipped whereas the custom solution uses xz (LZMA2), allowing for up to 70% smaller packages.
|
||||
|
||||
## How does it work
|
||||
|
||||
The package manager is integrated into the VMMgr application. It can be invoked only via the VMMgr web GUI. The entry point is on `/setup-apps` URL and all the configuration related to the repository settings (URL, username and password) can be configured on the same page. The URL should point to the directory where all content previously created by repository maintainer using `lxc-pack` commands is uploaded (i.e. `packages`, `packages.sig` and all `*.tar.xz` files). Once the user opens this page, VMMgr tries to contact the repository using the configured values and attempts to download `packages` file with all packages metadata. If it fails, it checks the reason for the failure (either connection exception or HTTP status code) and displays appropriate error message to the user.
|
||||
|
||||
If the `packages` is successfully downloaded, the package manager immediately downloads also `packages.sig`, which contains ECDSA-signed SHA512 hash of the `packages` file, and verifies the signature using public key preloaded in `/etc/vmmgr/packages.pub`. In the signature matches, then it parses the `packages` file contents in JSON format and displays list of installable packages.
|
||||
|
||||
The information about installed packages, including their metadata, are stored in a local metadata file `/etc/vmmgr/config.json` along with VMMgr settings for the local virtual machine. The local metadata file is also in JSON format and the metadata are simply copied to it from the remote repository during installation.
|
||||
|
||||
All package manager actions (install / upgrade / uninstall) as well as stop / start actions are handled by VMMgr queue manager. The queue manager processes the actions sequentially in the order in which they were enqueued (FIFO), therefore it cannot happen that multiple package installations will run simultaneously or will interfere with stop / start actions. In the event of unexpected failure or VM shutdown, it is possible to safely repeat the failed or unfinished actions as the install / upgrade / uninstall methods are designed to ensure sanity of the environment.
|
||||
|
||||
The whole idea is generally the same as with any other packaging system - e.g. *rpm* or *dpkg* on linux, *homebrew* on Mac or *Chocolatey* on Windows, except this packaging system is highly customised for use with LXC containers and VMMgr web GUI.
|
||||
|
||||
### Anatomy of a package
|
||||
|
||||
The files in the package are structured as
|
||||
|
||||
```
|
||||
*.tar.xz
|
||||
├─ srv/
|
||||
│ └─ <package>/
|
||||
│ ├─ install/
|
||||
│ ├─ install.sh
|
||||
│ ├─ uninstall/
|
||||
│ ├─ uninstall.sh
|
||||
│ ├─ upgrade/
|
||||
│ └─ upgrade.sh
|
||||
└─ var/
|
||||
└─ lib/
|
||||
└─ lxc/
|
||||
└─ <lxcpath>/
|
||||
```
|
||||
|
||||
This structure is extracted directly to the root directory of the virtual machine as it would be with any other package manager. Every package may contain subdirectories `install`, `upgrade` and `uninstall` and files `install.sh`, `upgrade.sh`, `uninstall.sh` which are invoked during the respective actions. Their presence and the contents under `/var/lib/lxc` depend on the type of the package. If the package contains only a shared LXC OverlayFS layer, it doesn't contain `config` file with LXC container definition and it likely wouldn't contain any of the install / upgrade / uninstall scripts and directories as it is not needed in this context.
|
||||
|
||||
### Installing a package
|
||||
|
||||
First, the installation method builds and flattens a dependency tree using the metadata from the repository and compares it with list of currently installed packages taken from the local metadata file, resulting in a list of packages to be downloaded and installed, ordered by dependency requirements (i.e. packages with already satisfied dependencies are installed first, satisfying dependencies for the subsequent ones).
|
||||
|
||||
All packages in this list are then downloaded as `*.tar.xz` archives from the repository and stored in temporary directory `/var/cache/vmmgr` as `*.tar.xz.partial`. Once the package is downloaded, its SHA512 has is calculated and verified against the value in cryptographically signed `packages` metadata file. If the hashes don't match, the whole installation process is interrupted and an error message informing about the mismatch is displayed to user. If the hashes match, the `*.tar.xz.partial` is renamed as `*.tar.xz`. Therefore in the event of unexpected VM shutdown or connection interruption, all `*.tar.xz` archives in `/var/cache/vmmgr` can be considered as verified and don't need to be downloaded again when the user decides to retry the installation.
|
||||
|
||||
Once all the packages are downloaded and their checksums are verified, the installation method unpacks them. Prior to unpacking, the method ensures sanity of filesystem by purging the directories and files (if they exist) which are to be used by the currently installing packages. This includes `/var/lib/lxc/<lxcpath>`, `/srv/<package>` and `/var/log/lxc/<package>.log`. The `*.tar.xz` archive is deleted right after decompressing.
|
||||
|
||||
After all the package archives are unpacked, the `uninstall.sh` script is run (if it is present) to ensure sanity of other components. This script attempts to remove objects and interfaces installed within components which are not part of the currently installing package (databases and database users, Solr cores, MQ definitions...). This requires that the `uninstall.sh` script is written in a defensive manner (e.g. `DROP DATABASE IF EXISTS...`)and must not exit with non-zero code even if no objects and interfaces for this package exist yet.
|
||||
|
||||
Next, an `install.sh` script is run which sets all the objects and interfaces which need to be installed in other components (databases, database users) and performs all the post-installation steps for the currently installing package, such as creation of persistent configuration and data directory under `/srv/<package>` of the VM. In case of user-installable application packages, the very last command in the `install.sh` script is `vmmgr register-app` [VMMgr hook](vmmgr-hooks) which creates a definition for VMMgr web GUI, including administrator credentials and subdomain of which the application will be accessible.
|
||||
|
||||
Finally the package itself with its metadata, stripped of `size` and `sha512` keys automatically added by `lxc-pack` during packaging, is added to the local repository metadata in `/etc/vmmgr/config.json`. After this, the package is considered fully installed and can be used by the users or other applications.
|
||||
|
||||
### Upgrading a package
|
||||
|
||||
Upgrading process is not yet implemented in the package manager. The idea is that the VMMgr simply compares the version and release from the repository metadata with the local metadata and offers upgrade if the version don't match. The dependency list build, download and verification part will be the same as during installation. Upgrade process will purge only the LXC data and LXC log, but will leave configuration and data under `/srv/<package>` unchanged. Then it overwrites install / upgrade / uninstall scripts and directories and runs `upgrade.sh` script. Finally it re-registers the package metadata in local repository metadata file `/etc/vmmgr/config.json`.
|
||||
|
||||
### Uninstalling a package
|
||||
|
||||
Uninstallation process first compiles a dependency list in a similar fashion like in the first step of installation, except this time it checks which packages are recorded as dependencies and will become unused (and therefore unnecessary) after the current package is uninstalled.
|
||||
|
||||
For every package in this list the `uninstall.sh` script is run, removing objects and interfaces installed within components which are not part of the currently installing package (databases and database users, Solr cores, MQ definitions...).
|
||||
|
||||
After the `uninstall.sh` finishes, all files related to the currently uninstalling package are deleted. This includes `/var/lib/lxc/<lxcpath>`, `/srv/<package>` and `/var/log/lxc/<package>.log`.
|
||||
|
||||
As the final step, the package metadata are unregistered (removed) from the local repository metadata file `/etc/vmmgr/config.json`.
|
||||
179
doc/toolchain/spoc-architecture.rst
Normal file
179
doc/toolchain/spoc-architecture.rst
Normal file
@ -0,0 +1,179 @@
|
||||
SPOC architecture
|
||||
=================
|
||||
|
||||
Configuration file
|
||||
------------------
|
||||
|
||||
The configuration file is located in ``/etc/spoc/spoc.conf`` and contains just a few directives. Standard user rarely needs to touch it, builder may need to adjust a few values for the building and package signing to work.
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[general]
|
||||
data-dir = /var/lib/spoc/
|
||||
log-dir = /var/log/spoc/
|
||||
network-interface = spocbr0
|
||||
|
||||
[publish]
|
||||
publish-dir = /srv/build/spoc/
|
||||
signing-key = /etc/spoc/publish.key
|
||||
|
||||
[repo]
|
||||
url = https://repo.spotter.cz/spoc/
|
||||
public-key = MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEWJXH4Qm0kt2L86sntQH+C1zOJNQ0qMRt0vx4krTxRs9HQTQYAy//JC92ea2aKleA8OL0JF90b1NYXcQCWdAS+vE/ng9IEAii8C2+5nfuFeZ5YUjbQhfFblwHSM0c7hEG
|
||||
|
||||
Section general contains common settings which is absolutely necessary for SPOC to work.
|
||||
|
||||
- ``data-dir`` is the path to the root directory of SPOC, under which all data are stored - repository metadata, layers, container configurations, persistent volumes, installation scripts etc. See section *Filesystem structure* for details on hierarchy.
|
||||
- ``log-dir`` is the path to log directory, where transcripts of PTYs of the individual containers are logged.
|
||||
- ``network-interface`` is the host bridge interface dedicated for use with SPOC containers. There are some assumptions mate to make the configuration as lightweight as possible. See section *Networking* for more details.
|
||||
|
||||
Section publish contains configuration for image and application building and publishing. Consumer doesn't need to set anything in this section.
|
||||
|
||||
- ``publish-dir`` is the path to which the newly built containers and applications will be published, i.e. where the archive and metadata will be stored and can be picked up manually od by another process to be transferred elsewhere.
|
||||
- ``signing-key`` is the path to ECDSA private key used for digitally signing the built packages. See section `Package manager` on more details about how are the packages signed and verified.
|
||||
|
||||
Section repo contains information about online repository used by the consumer to access the metadata and download the packages.
|
||||
|
||||
- ``url`` is the URL of the base repository directory. The URL may contain username and password and are expected to be in RFC 1738 compliant common internet scheme syntax ``<scheme>://<user>:<password>@<host>:<port>/<url-path>``.
|
||||
- ``public-key`` is the pinned ECDSA public key used for verification of metadata and downloaded content. See section `Package manager` on more details about how are the packages signed and verified.
|
||||
|
||||
Package manager
|
||||
---------------
|
||||
|
||||
There are several types of deliverables, however only images and application are subject to packaging and lifecycle management. In order to create a deliverable package, the builder must have an ECDSA private key, which will be used for signing. The principle is similar to any other public-key cryptography, so an RSA or GPG keys could have been used, but ECDSA has been chosen as it uses has reasonably short keys while retaining high level of security.
|
||||
|
||||
When an image or application are packaged, the files (the layer in case of image and the setup scripts in case of application) are archived as *.tar.xz*, placed in the publish repository. XZ (LZMA2) has been selected as it has the best compression ratio from the commonly used compression algorithms. Then, using the builders ECDSA private key, a signed SHA512 hash of the archive is calculated and stored with the rest of the metadata. This hash, uncompressed data size and compressed archive size (download size) are written into the metadata file along with the rest of the metadata. Finally, another signed SHA512 hash is calculated from the metadata JSON file ``repository.json`` and stored as a separate signature file ``repository.sig``. All files (XZ archives, repository.json and repository.sig) then need to be transferred to a desired location on a distribution server.
|
||||
|
||||
In order to consume the packages, the consumer needs to know the builder's, resp. online repository's ECDSA public key and needs to have it configured (pinned) in ``/etc/spoc/spoc.conf``, otherwise any attempt for signature verification will fail.
|
||||
|
||||
The consumer issues a command to install the application. First, SPOC downloads the ``repository.json`` metadata file and the ``repository.sig`` signature file and verifies the signature of the metadata file. If the signature can't be verified, installation will immediately fail at this point. Otherwise, the installation continues. SPOC's dependency resolver then checks which images and layers does the application (resp. its containers) require and compares the requirements with the packages already registered in the local repository. If the application requires any images which are not yet locally available, they will be downloaded. The package is downloaded into a temporary location under ``/var/lib/spoc/tmp``. All downloaded archives have their signature verified against the signed hash contained in ``repository.json``. This ensures both that the archive has been downloaded completely and without errors and that it hasn't been tampered with. Only after a successful verification it is extracted. The extraction reuses the same file descriptor to make the verification and extraction operation atomic in attempt to prevent any local attack vectors and replacements of the archive between the two subtasks. Once the image archive is unpacked, its metadata are registered to the local repository and the downloaded archive is deleted.
|
||||
|
||||
When all required images are present, the archive with application setup scripts is downloaded, verified and unpacked in similar fashion. Then the containers are created as described in metadata. If an ``uninstall.sh`` file is present in the application archive, it is run in order to ensure sanity of the environment. The script is supposed to clean up any data from any previous potentially failed installations. Then, if ``install.sh`` script is present in the now extracted application archive, it is executed. It is expected that it will prepare the application environment for immediate usage by the consumer (configures the application, generates passwords, populates database etc.). If the script fails, the containers are attempted to be destroyed and cleaned up, so the installation can be retried at a future time. If the installation succeeds, the application is registered to the local repository.
|
||||
|
||||
SPOC compares the ``"version"`` field using `semantic versioning <https://semver.org/>`_ rules. Once a new version of application is packages and made available in the online repository, the consumer may choose to update the locally installed version. All containers belonging to the application will be stopped and the dependency solver will check if any new images/layers are to be downloaded. After all necessary packages are downloaded, verified and unpacked, SPOC destroys all containers existing in the previous application versions and recreates new ones for the current version. The recreated containers are expected to reuse the persistent storage for application data from the previous version. After the containers are recreated, an ``update.sh`` script is launched, if it exists in the application archive. This script carries out any update or migration steps to make the application environment ready for immediate usage by the consumer after the upgrade. Old images/layers which are no longer in use are not automatically removed, but the user may issue ``spoc-image clean`` cleanup command to do so.
|
||||
|
||||
When the consumer wishes to uninstall the application, SPOC destroys the containers including their log files and runs the ``uninstall.sh`` application script, if it is present. The script is expected to clean up persistent volumes for the application. Finally, SPOC unregisters the application from the local repository metadata and removes its setup scripts. Once again, the orphaned images/layers are not automatically removed, but the user may do so with ``spoc-image clean`` cleanup command.
|
||||
|
||||
Filesystem structure
|
||||
--------------------
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
/etc/spoc/
|
||||
├─ spoc.conf - Main (and the only) configuration file described above
|
||||
└─ publish.key - ECDSA private key used by the builder for package signing
|
||||
|
||||
/var/lib/spoc/ - Root directory with all data
|
||||
├─ apps/ - Applications directory
|
||||
│ └─ <application>/ - Application-specific directory with install/update/uninstall scripts
|
||||
├─ containers/ - Containers directory
|
||||
│ └─ <container>/ - Container specific data
|
||||
│ ├─ config - LXC container configuration file
|
||||
│ ├─ ephemeral/ - Directory with topmost ephemeral OverlayFS layer
|
||||
│ ├─ olwork/ - Working directory for OverlayFS
|
||||
│ └─ rootfs/ - Mountpoint for the OverlayFS serving as the root directory for the container
|
||||
├─ hosts - SPOC-global /etc/hosts file shared by all containers
|
||||
├─ layers/ - Layers directory
|
||||
│ └─ <layer> - Layer-specific files
|
||||
├─ repository.json - Local repository containing info about installed images, containers and apps
|
||||
├─ tmp/ - Directory where the downloaded archives are temporarily stored before they are unpacked
|
||||
│ ├─ apps/ - Temporary location for application archives
|
||||
│ └─ layers/ - Temporary location for layer archives
|
||||
└─ volumes/ - Volumes directory
|
||||
└─ <volume>/ - Specific persistent volume storage
|
||||
|
||||
/var/lock/ - Standard system directory with locks
|
||||
├─ spoc-hosts.lock - Lock file for operations with SPOC-global hosts file
|
||||
├─ spoc-local.lock - Lock file for operations with local repository metadata file
|
||||
├─ spoc-publish.lock - Lock file for operations with publishing repository metadata file
|
||||
└─ spoc.lock - Main lock file for the installation operations
|
||||
|
||||
/var/log/spoc/ - Log directory
|
||||
└─ <container>.log - Container-specific PTY transcript log file
|
||||
|
||||
Repository metadata
|
||||
-------------------
|
||||
|
||||
Metadata are stored as JSON file. They contain all application, container and image definitions and the relations between them. The metadata basically contain all the information besides the actual files used by OverlayFS.
|
||||
|
||||
Metadata pertaining to local installation reside in ``/var/lib/spoc/repository.json`` and look as follows.
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"apps": {
|
||||
"<application>": {
|
||||
"containers": [
|
||||
"<container>"
|
||||
],
|
||||
"meta": {
|
||||
"desc-cs": "<Description in Czech>",
|
||||
"desc-en": "<Description in English>",
|
||||
"license": "<Licence type>",
|
||||
"title": "<Application Title>"
|
||||
},
|
||||
"version": "<version>"
|
||||
}
|
||||
},
|
||||
"containers": {
|
||||
"<container>": {
|
||||
"cmd": "<init command>",
|
||||
"gid": "<gid>",
|
||||
"layers": [
|
||||
"<layer>"
|
||||
],
|
||||
"mounts": {
|
||||
"<volume>": "<mountpoint>"
|
||||
},
|
||||
"ready": "<readiness check command>",
|
||||
"uid": "<uid>"
|
||||
}
|
||||
},
|
||||
"images": {
|
||||
"<image>": {
|
||||
"cmd": "<init command>",
|
||||
"dlsize": "<archive size>",
|
||||
"hash": "<archive hash>",
|
||||
"layers": [
|
||||
"<layer>"
|
||||
],
|
||||
"size": "<filesystem size>"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Networking
|
||||
----------
|
||||
|
||||
Container managers traditionally use services like *dnsmasq* to provide DNS and DHCP capabilities. SPOC, however, uses very naive approach and use mostly static network configuration. SPOC expects that simple init systems will be used in the containers and that the network configuration can be done in a "traditional" way via using static IP addresses, ``/etc/hosts`` and ``/etc/resolv.conf`` files.
|
||||
|
||||
The cornerstone of network configuration is the host bridge. The default host bridge found on VM is named ``spocbr0`` and has IP 172.17.0.1/16. The IP and netmask is what SPOC uses for configuration, so just from the IP and mask, it infers that the containers need to have their IP addresses in range 172.17.0.2 - 172.17.255.254 with the same netmask.
|
||||
|
||||
*Poor man's DHCP* is implemented via SPOC-global hosts file ``/var/lib/spoc/hosts``. This file holds the IP leases for the individual containers and at the same time it is readonly-mounted in the containers as ``/etc/hosts``, so the containers have the information about surrounding containers and can communicate with them. Leases are added when a container is created and removed when the container is destroyed, so the container has the same IP address for its whole lifetime.
|
||||
|
||||
Since there is no DNS server, ``/etc/resolv.conf`` of the host is readonly-mounted into the container as-is. The containers therefore inherit the DNS configuration of the host this way.
|
||||
|
||||
File locking
|
||||
------------
|
||||
|
||||
SPOC uses file locks for every operation where there is a risk of a race condition while writing metadata or other files. This allows for simultaneous run of multiple install / update / uninstall commands which will always race for the lock and will therefore be run in sequence without any interferences. The individual actions needed for the operation to succeed are always determined when the process performing the operation obtains the lock and starts processing it. Whenever another tool uses the SPOC API (python module), it is important to use the locking mechanism as well. VMMgr uses the locking, so it's safe to run install / update / uninstall from command line and web UI in parallel. They will be automatically serialized using the lock.
|
||||
|
||||
OverlayFS hooks
|
||||
---------------
|
||||
|
||||
The containers are run as unprivileged ones, but they are started by the root user. This helps to work around some restrictions of OverlayFS mounts and simplifies changing ownership of the files. Normally, OverlayFS mounting is restricted to root and it's `not possible to create OverlayFS mount from user namespace <https://www.spinics.net/lists/linux-fsdevel/msg105877.html>`_. Some linux distributions, notably Debian and Ubuntu, ship with kernel `patches allowing to circumvent the restriction <https://salsa.debian.org/kernel-team/linux/blob/master/debian/patches/debian/overlayfs-permit-mounts-in-userns.patch>`_, however this is not the case with Alpine. Another option is to use `fuse-overlayfs <https://github.com/containers/fuse-overlayfs>`_, but it has some known problems on non-glibc linuxes and is not mature enough for production usage.
|
||||
|
||||
SPOC therefore simply uses the standard OverlayFS mounts, however since the containers are unprivileged and the rootfs mounting traditionally happens in the container namespace, the overlay layers can't be defined in the LXC container configuration. Instead, they are taken from the local repository metadata and the rootfs mountpoint is prepared before the container namespace, using LXC hooks.
|
||||
|
||||
During the container startup ``lxc.hook.pre-start`` event hook launches a process which will clean the contents of the topmost ephemeral layer in case the container has been shut down unexpectedly before, and creates the rootfs via standard means.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mount -t overlay -o upperdir=/var/spoc/lib/containers/<container>/ephemeral,lowerdir=/var/spoc/lib/layers/<layer1>:/var/spoc/lib/layers/<layer2>,workdir=/var/spoc/lib/containers/<container>/olwork none /var/spoc/lib/containers/<container>/rootfs
|
||||
|
||||
On container shutdown, ``lxc.hook.post-stop`` event hook launches a process again which unmounts the rootf and cleans the ephemeral layer.
|
||||
|
||||
Init service
|
||||
------------
|
||||
|
||||
SPOC install an init service, however no long-lived service process is run. The init task simply executes ``spoc-app start-autostarted`` on host startup and ``spoc-app stop-all`` on host shutdown. The former command starts all containers for all application which have autostart flag set, the latter will stop all currently running SPOC containers.
|
||||
371
doc/toolchain/spoc-builder.rst
Normal file
371
doc/toolchain/spoc-builder.rst
Normal file
@ -0,0 +1,371 @@
|
||||
SPOC building and packaging
|
||||
=============================
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
``spoc-image build`` creates an LXC image based on a file with a sequence of build directives (also known as build recipe) given via command line parameter. The directory in which the file resides is taken as build context, i.e. all relative paths in the recipe are resolved from it.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image build [-f] [-p] filename
|
||||
|
||||
positional arguments:
|
||||
filename Path to the file with build recipe
|
||||
|
||||
optional arguments:
|
||||
-f, --force Force rebuild already existing image
|
||||
-p, --publish Publish the image after successful build
|
||||
|
||||
``spoc-image publish`` creates a `.tar.xz` archive of the built image and publishes it to the publish repository along with its metadata.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image publish [-f] image
|
||||
|
||||
positional arguments:
|
||||
image Name of the image to publish
|
||||
|
||||
optional arguments:
|
||||
-f, --force Force republish already published image
|
||||
|
||||
To remove published image from publish repo, ``spoc-image unpublish`` command should be used.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image unpublish image
|
||||
|
||||
positional arguments:
|
||||
image Name of the image to unpublish
|
||||
|
||||
Conversely, ``spoc-app publish`` creates a `.tar.xz` archive of the application setup scripts (``install.sh``, ``update.sh``, ``uninstall.sh`` and associated directories) and publishes it to the publish repository along with its metadata.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app publish [-f] filename
|
||||
|
||||
positional arguments:
|
||||
filename Path to metadata file of the application to publish
|
||||
|
||||
optional arguments:
|
||||
-f, --force Force republish already published application
|
||||
|
||||
And to remove the published application, ``spoc-app unpublish`` should be used.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app unpublish [-h] app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to unpublish
|
||||
|
||||
Build directives
|
||||
----------------
|
||||
|
||||
The syntax is designed to resemble *Dockerfile* syntax in order to ease the potential transition. Since LXC operates on much lower level of abstraction than Docker, some principles are applied more explicitly and verbosely. Major difference between Docker and SPOC is that every directive in *Dockerfile* creates a new filesystem layer whereas layers in SPOC are managed manually and there's usually only a single layer produced from the build recipe.
|
||||
|
||||
IMAGE
|
||||
^^^^^
|
||||
|
||||
- **Usage:** ``IMAGE <name>``
|
||||
- **Description:** Sets image/layer name. Every image/layer needs to have one. Any subsequent directives requiring to be handled in containers namespace will use this name also to create a temporary LXC container.
|
||||
- **Docker equivalent:** ``-t`` in ``docker build`` command line parameters
|
||||
|
||||
FROM
|
||||
^^^^^
|
||||
|
||||
- **Usage:** ``FROM <path>``
|
||||
- **Description:** Designates an OverlayFS layer on which the layer will be based. Unlike *Dockerfile*'s `FROM`, is SPOC this directive is optional, so if no ``FROM`` directive is given, it translates to Docker's ``FROM scratch``.
|
||||
- **Docker equivalent:** ``FROM``
|
||||
|
||||
RUN
|
||||
^^^
|
||||
|
||||
- **Usage:**
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
RUN <label>
|
||||
<commands>
|
||||
<label>
|
||||
|
||||
- **Description:** Executes a shell script in the image/container which is currently being built. The ``<label>`` is an arbitrary user defined string which needs to be given as the first parameter and repeated at the end of the script block, in similar fashion like heredoc label to which it actually translates. The shell script between the labels is passed as-is, including comments and empty lines, to a POSIX shell with ``-e`` and ``-v`` parameters set. Basically, following ``RUN`` entry:
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
RUN EOF
|
||||
# Comment
|
||||
command1
|
||||
command2
|
||||
EOF
|
||||
|
||||
translates to the following script:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#!/bin/sh
|
||||
set -ev
|
||||
|
||||
# Comment
|
||||
command1
|
||||
command2
|
||||
|
||||
The command chaining via ``&&`` which is required in *Dockerfile* is not required in SPOC.
|
||||
|
||||
- **Docker equivalent:** ``RUN``
|
||||
|
||||
COPY
|
||||
^^^^
|
||||
|
||||
- **Usage:** ``COPY <source> [destination]``
|
||||
- **Description:** Recursively copies ``<source>`` files into ``<destination>``. Source path is relative to the build context directory, destination path is relative to the container root directory. The files are copied with the same permissions and owner/group, which are immediately shifted to the appropriate UID/GID within the container namespace. The ``<source>`` can be given as *http://* or *https://* URL in which case gzip, bzip2 or xz tar archive is expected to be downloaded and unpacked into the ``<destination>``. This is commonly used for creating a basic root filesystem of the container in similar fashion like with Docker's ``FROM scratch``.
|
||||
- **Docker equivalent:** ``COPY`` or ``ADD``
|
||||
|
||||
USER
|
||||
^^^^
|
||||
|
||||
- **Usage:** ``USER <uid> <gid>``
|
||||
- **Description:** Sets UID/GID of the container init process to ``<uid>`` and ``<gid>``. The default UID/GID is ``0:0 (root:root)``. The values can be given also as user/group name in which case they're looked up in the image/container namespace.
|
||||
- **Docker equivalent:** ``USER``
|
||||
- **Populates LXC field:** ``lxc.init.uid`` and ``lxc.init.gid``
|
||||
|
||||
CMD
|
||||
^^^
|
||||
|
||||
- **Usage:** ``CMD <command> [parameters...]``
|
||||
- **Description:** Sets the init process of the container. This is the process which is automatically started after the container is launched. The default command is ``/bin/true`` which immediately terminates with return code 0.
|
||||
- **Docker equivalent:** ``CMD``
|
||||
- **Populates LXC field:** ``lxc.init.cmd``
|
||||
|
||||
ENV
|
||||
^^^
|
||||
|
||||
- **Usage:** ``ENV <variable> <value>``
|
||||
- **Description:** Populates environment variable ``<variable>`` with value ``<value>`` which is then passed to the init process when the container is launched and is expected to be propagated by it to subsequently launched processes.
|
||||
- **Docker equivalent:** ``ENV``
|
||||
- **Populates LXC field:** ``lxc.environment``
|
||||
|
||||
WORKDIR
|
||||
^^^^^^^
|
||||
|
||||
- **Usage:** ``WORKDIR <dirname>``
|
||||
- **Description:** Sets working directory of the container init process to ``<dirname>``. The default working directory is the container's root directory.
|
||||
- **Docker equivalent:** ``WORKDIR``
|
||||
- **Populates LXC field:** ``lxc.init.uid`` and ``lxc.init.gid``
|
||||
|
||||
HALT
|
||||
^^^^
|
||||
|
||||
- **Usage:** ``HALT <signal>``
|
||||
- **Description:** Sets container stop signal to ``<signal>``. The default signal is SIGINT.
|
||||
- **Docker equivalent:** ``--signal`` in ``docker kill`` command line parameters
|
||||
- **Populates LXC field:** ``lxc.signal.halt``
|
||||
|
||||
READY
|
||||
^^^^^
|
||||
|
||||
- **Usage:** ``READY <command> [parameters...]``
|
||||
- **Description:** Sets a command to be run after the container is started to check if the container is ready and a depending container may be started too. The command is retried until it returns returncode 0 or until the attempt to start the container times out after 30 seconds.
|
||||
- **Docker equivalent:** ``HEALTHCHECK``
|
||||
|
||||
LXC config
|
||||
----------
|
||||
|
||||
Although SPOC populates some LXC config fields, there are lot of defaults with remain unchanged. The template file to which *spoc-image build* fills in the values looks as follows:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
# Container name
|
||||
lxc.uts.name = {name}
|
||||
|
||||
# Network
|
||||
lxc.net.0.type = veth
|
||||
lxc.net.0.link = spocbr0
|
||||
lxc.net.0.flags = up
|
||||
lxc.net.0.ipv4.address = {ip_address}/{ip_netmask}
|
||||
lxc.net.0.ipv4.gateway = {ip_gateway}
|
||||
|
||||
# Root filesystem
|
||||
lxc.rootfs.path = {rootfs}
|
||||
|
||||
# Mounts
|
||||
lxc.mount.entry = shm dev/shm tmpfs rw,nodev,noexec,nosuid,relatime,mode=1777,create=dir 0 0
|
||||
lxc.mount.entry = /etc/resolv.conf etc/resolv.conf none bind,ro,create=file 0 0
|
||||
lxc.mount.entry = {hosts} etc/hosts none bind,ro,create=file 0 0
|
||||
{mounts}
|
||||
|
||||
# Environment
|
||||
lxc.environment = PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
{env}
|
||||
|
||||
# Init
|
||||
lxc.init.uid = {uid}
|
||||
lxc.init.gid = {gid}
|
||||
lxc.init.cwd = {cwd}
|
||||
lxc.init.cmd = {cmd}
|
||||
|
||||
# Halt
|
||||
lxc.signal.halt = {halt}
|
||||
|
||||
# Log
|
||||
lxc.console.size = 1MB
|
||||
lxc.console.logfile = {log}
|
||||
|
||||
# ID map
|
||||
lxc.idmap = u 0 100000 65536
|
||||
lxc.idmap = g 0 100000 65536
|
||||
|
||||
# Hooks
|
||||
lxc.hook.version = 1
|
||||
lxc.hook.pre-start = /usr/bin/spoc-hook
|
||||
lxc.hook.post-stop = /usr/bin/spoc-hook
|
||||
|
||||
# Other
|
||||
lxc.arch = linux64
|
||||
lxc.include = /usr/share/lxc/config/common.conf
|
||||
lxc.include = /usr/share/lxc/config/userns.conf
|
||||
|
||||
For explanation of hooks and overall container integration and behavior, refer to `OverlayFS hooks <spoc-architecture>`_ section on `SPOC Architecture <spoc-architecture>`_ page.
|
||||
|
||||
Example build recipe
|
||||
--------------------
|
||||
|
||||
The following is an example of build recipe for *Redis* image:
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
IMAGE redis_5.0.7-200403
|
||||
FROM alpine3.11_3.11.5-200403
|
||||
|
||||
RUN EOF
|
||||
# Create OS user (which will be picked up later by apk add)
|
||||
addgroup -S -g 6379 redis
|
||||
adduser -S -u 6379 -h /var/lib/redis -s /bin/false -g redis -G redis redis
|
||||
|
||||
# Install Redis
|
||||
apk --no-cache add redis
|
||||
EOF
|
||||
|
||||
USER redis
|
||||
CMD /usr/bin/redis-server /etc/redis.conf
|
||||
|
||||
Container interfaces
|
||||
--------------------
|
||||
|
||||
Due to the fact that LXC container has all environment variables, mounts, used layers, init and everything it needs for starting, running and stopping in its configuration file and SPOC metadata, you are completely free to build the container in any way possible. The only requirement imposed by the host (virtual machine) infrastructure is that the container has to be a linux one.
|
||||
|
||||
There's currently another restriction for containers with user-accessible web interfaces via HTTP(S) as the requests need to be reverse-proxied via VM's nginx HTTP server. The application within the container needs to be reachable via plain HTTP on port 8080/TCP. The reverse proxy sets the ``X-Forwareded-*`` headers which can (and should) be processed by the application.
|
||||
|
||||
Application publishing
|
||||
----------------------
|
||||
|
||||
Application is defined by a JSON file with metadata and optionally by installation, update and uninstallation scripts and files associated to them.
|
||||
|
||||
- ``"version": "<version string>"`` - Defines the version of application and the value from local repository is compared with the one in online repository whenever the consumer checks for updated.
|
||||
- ``"meta": { "<key>": "value" }`` - Attaches arbitrary metadata intended for consumption by other tools, e.g. VMMgr. These metadata have no meaning for SPOC.
|
||||
- ``"containers": { "<name>": {<container definition>} }`` - Creates a definition for container. More directives for container definitions continue below.
|
||||
- ``"image": "<image name>"`` - Defines which image is to be used as the base for the container creation.
|
||||
- ``"depends": [ "<container name>" ]`` - Defines start dependencies between containers. This helps to ensure that e.g. a database container will be always ready before the application container is attempted to be started.
|
||||
- ``"mounts": { "<volume path>": "<mountpoint>" }`` - Defines persistent volumes and their mountpoints within the container. The volume path is a relative path under ``/var/lib/spoc/volumes``. The mountpoint is a relative path under container's rootfs. If the mountpoint doesn't exist within the container, it is created. By default, the volume and the mountpoint are directories, but optionally, the mountpoint can be suffixed with ``:file`` which instructs SPOC to handle the paths as regular files instead of directories.
|
||||
- ``"env": { "<key>": "<value>" }`` - Defines environment variables specific for the container. These variables are set when the init process starts and it's up to the init process to ensure any propagation to the target applications.
|
||||
|
||||
Setup scripts
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
While packaging / publishing, SPOC looks into the directory with the JSON file and searches for files called ``install.sh``, ``update.sh``, ``uninstall.sh`` and directories named ``install``, ``update`` and ``uninstall``, when the appropriately named script is found. These scripts and directories carry the setup steps, data, configuration files and templates required for the application to be fully installed and used by the consumer without any additional configuration.
|
||||
|
||||
The scripts need to be executable and can be written in any language, however traditionally they are written as shell scripts. Note that the VM has ony POSIX-compliant *ash* shell, not *bash*.
|
||||
|
||||
The script is always executed with working directory set to the directory where it resides. All path to the installation files can then be put as relative paths. If the need arises, the scripts can rely on following environment variables passed by SPOC, in case the default data path is not ``/var/lib/spoc``.
|
||||
|
||||
- ``LAYERS_DIR`` - Directory where the layers are stored. ``/var/lib/spoc/layers`` by default.
|
||||
- ``VOLUMES_DIR`` - Directory where the persistent volumes are stored. ``/var/lib/spoc/volumes`` by default.
|
||||
- ``APPS_DIR`` - Directory where the application scripts are stored. ``/var/lib/spoc/apps`` by default.
|
||||
- ``LOG_DIR`` - Directory with container logs. ``/var/log/spoc`` by default.
|
||||
|
||||
Example application metadata
|
||||
----------------------------
|
||||
|
||||
The following is an example of application metadata for *Kanboard* application:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"version": "1.2.14-200416",
|
||||
"meta": {
|
||||
"title": "Kanboard",
|
||||
"description": "Kanban project management",
|
||||
"license": "GPL"
|
||||
},
|
||||
"containers": {
|
||||
"kanboard": {
|
||||
"image": "kanboard_1.2.14-200416",
|
||||
"depends": [
|
||||
"kanboard-postgres"
|
||||
],
|
||||
"mounts": {
|
||||
"kanboard/kanboard_data": "srv/kanboard/data/files",
|
||||
"kanboard/kanboard_conf/config.php": "srv/kanboard/config.php:file"
|
||||
}
|
||||
},
|
||||
"kanboard-postgres": {
|
||||
"image": "postgres_12.2.0-200403",
|
||||
"mounts": {
|
||||
"kanboard/postgres_data": "var/lib/postgresql"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
And the associated ``install.sh`` script (other files mentioned in the script are not shown)
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#!/bin/sh
|
||||
set -ev
|
||||
|
||||
# Volumes
|
||||
POSTGRES_DATA="${VOLUMES_DIR}/kanboard/postgres_data"
|
||||
KANBOARD_CONF="${VOLUMES_DIR}/kanboard/kanboard_conf"
|
||||
KANBOARD_DATA="${VOLUMES_DIR}/kanboard/kanboard_data"
|
||||
|
||||
# Create Postgres instance
|
||||
install -o 105432 -g 105432 -m 700 -d ${POSTGRES_DATA}
|
||||
spoc-container exec kanboard-postgres -- initdb -D /var/lib/postgresql
|
||||
|
||||
# Configure Postgres
|
||||
install -o 105432 -g 105432 -m 600 postgres_data/postgresql.conf ${POSTGRES_DATA}/postgresql.conf
|
||||
install -o 105432 -g 105432 -m 600 postgres_data/pg_hba.conf ${POSTGRES_DATA}/pg_hba.conf
|
||||
|
||||
# Configure Kanboard
|
||||
export KANBOARD_PWD=$(head -c 18 /dev/urandom | base64 | tr -d '+/=')
|
||||
install -o 108080 -g 108080 -m 750 -d ${KANBOARD_CONF}
|
||||
install -o 108080 -g 108080 -m 750 -d ${KANBOARD_DATA}
|
||||
envsubst <kanboard_conf/config.php | install -o 108080 -g 108080 -m 640 /dev/stdin ${KANBOARD_CONF}/config.php
|
||||
|
||||
# Populate database
|
||||
spoc-container start kanboard-postgres
|
||||
envsubst <createdb.sql | spoc-container exec kanboard-postgres -- psql
|
||||
spoc-container exec kanboard -- cat /srv/kanboard/app/Schema/Sql/postgres.sql | spoc-container exec kanboard-postgres -- sh -c "PGPASSWORD=${KANBOARD_PWD} psql kanboard kanboard"
|
||||
|
||||
# Create admin account
|
||||
export KANBOARD_ADMIN_USER="admin"
|
||||
export KANBOARD_ADMIN_PWD=$(head -c 12 /dev/urandom | base64 | tr -d '+/=')
|
||||
export KANBOARD_ADMIN_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw('${KANBOARD_ADMIN_PWD}'.encode(), bcrypt.gensalt()).decode().replace('2b', '2y'))")
|
||||
envsubst <adminpwd.sql | spoc-container exec kanboard-postgres -- psql kanboard
|
||||
|
||||
# Stop services required for setup
|
||||
spoc-container stop kanboard-postgres
|
||||
|
||||
Recommended tools and practices
|
||||
-------------------------------
|
||||
|
||||
If the application itself doesn't support connection via plain HTTP (e.g. it is a CGI/WSGI application), the container needs to contain also a web server which will proxy the connection. Recommended web server for this purpose is *nginx* HTTP server, which is lightweight and can server as proxy for all commonly used gateway interfaces. Bear in mind that in some cases, the application needs to able to infer its own HTTP host, so some header tuning for both HTTP and CGI/WSGI protocols might be in order.
|
||||
|
||||
In case there are more components or services running within the same container (e.g. nginx HTTP server and PHP-FPM), it is advised to have them spawned and supervised using some lightweight init daemon. Recommended init system for LXC containers is *s6*, however if you are familiar with *daemontools* or *runit*, feel free to use them as well. In the worst case you can use *OpenRC*. Systems like *Upstart*, *Systemd* or *SysV init* are not recommended for their complexity or inability to properly supervise the spawned processes.
|
||||
|
||||
When an image is built, its name (and name of its immediate layer) should be universally unique. The idea is that the images are not versioned (resp. SPOC is version agnostic towards images/layers), so depending on the application, resp. container requirements, it's entirely possible to have multiple "versions" of the same image/layer installed simultaneously. It's therefore required to have the version as part of the image/layer name. E.g. ``alpine3.11_3.11.5-200403`` - this makes it clear that it's an image/layer for Alpine 3.11, specifically 3.11.5 built on 2020-04-03. At the same time this makes it possible to have e.g. ``alpine3.11_3.11.5-200520`` (newer build of the same version, possibly containing some extra security fixes) simultaneously installed and used by another container without any conflicts between the two Alpine 3.11.5 layers.
|
||||
|
||||
The ``uninstall.sh`` script should be designed in a way which allows it to be successfully executed even when the application is not installed. The script is run *before* ``install.sh`` in order to clean up any data from any previous potentially failed installations.
|
||||
|
||||
Don't hardcode any absolute paths in the install / update / uninstall scripts. Always rely on ``LAYERS_DIR``, ``VOLUMES_DIR``, ``APPS_DIR`` and ``LOG_DIR`` environment variables.
|
||||
130
doc/toolchain/spoc-overview.rst
Normal file
130
doc/toolchain/spoc-overview.rst
Normal file
@ -0,0 +1,130 @@
|
||||
SPOC overview
|
||||
===============
|
||||
|
||||
SPOC is the main component of the platform. It provides means for the layers, images, containers and applications to be built, published, installed and run. The name "*SPOC*" may have many meanings - It can stand for **Spo**\ tterVM **C**\ ontainers, **S**\ ingle **P**\ oint **o**\ f **C**\ ontact, as it's the only component with which the builder and the user needs to interact, or it can represent a wish for the applications to "*Live long and prosper*" as is the catchphrase of the better-known bearer of the same name.
|
||||
|
||||
SPOC is written in python 3.7 and uses LXC as a userspace interface for the Linux kernel containment features. SPOC consists of a python module, configuration file, init task and three main binaries to work with the respective deliverable types.
|
||||
|
||||
Goals and non-goals
|
||||
-------------------
|
||||
|
||||
SPOC aims to:
|
||||
|
||||
- Be easily comprehensible from system administration point of view - the less abstraction the better
|
||||
- Be minimalistic - implements only features which are consumed locally
|
||||
- Be independent on any 3rd party services (both paid and free) or online repositories - everything is self-hosted
|
||||
- Have minimal storage size footprint for use on desktops and laptops and on connections with low bandwidth - prefer small size before speed
|
||||
|
||||
SPOC doesn't aim to:
|
||||
|
||||
- Replace or reimplement existing container management or application distribution solutions (Docker, Flatpak, Snap, Conda, BIBBOX etc.)
|
||||
- Implement any features for orchestration or remote management
|
||||
|
||||
Container runtime
|
||||
-----------------
|
||||
|
||||
LXC is used as the container runtime. There are several well-known container runtimes, with *Docker* being probably the most popular. At one point in the history of SpotterVM development, Docker has been considered and used, however its size requirements, opacity of configuration from administration point of view, and unsatisfactory overlap of required and provided functionality caused that LXC was eventually selected instead. This section should probably be more appropriately titled *Why not Docker* as most comparisons will be done to it.
|
||||
|
||||
First and foremost, Docker contains a huge set of tools for use with various orchestrators and large-scale applications. Docker is a perfect tool when multiple instances of the same container image need to run and controlled on multiple systems. The individual instances are configured on runtime via command line parameters or stored as Docker services in cluster configuration. Docker daemon and its shim processes contain a considerable amount of abstraction, effectively obstructing the visibility on what is happening under the hood. LXC, on the other hand, keeps things close to the bare minimum and transparently uses container techniques, syscalls and namespaces exposed by the linux kernel. The containers in LXC are fully defined via configuration files and don't require any additional configuration on runtime. This can arguably be achieved even on Docker via *docker-composer*, but that adds yet another layer of abstraction and generally is not suitable for scenarios where the container images need to be added or removed on the fly.
|
||||
|
||||
Docker is written in Go language, which is designed to create runtime-safe statically linked executables. With the shear amount of Docker capabilities, this unfortunately means that the whole Docker infrastructure occupies roughly 200 MB on the VM hard drive. The basic virtual machine image is designed to be as small as possible, so having a 200 MB large container host on an operating system which alone occupies roughly 40 MB does not seem ideal. LXC runtime written in C/C++ on the other hand occupies roughly 4 MB and doesn't need any other dependencies besides *cgroupfs* which, for performance reasons, is good to have installed anyway. This may change with general availability of Podman on Alpine linux, as Podman offers similar functionality as Docker in quarter of the size.
|
||||
|
||||
Due to the Docker's approach, storage overlay layers cannot be easily managed by the container builder. They instead depend on the amount and order of directives in *Dockerfile* recipe file. Docker layers reside in directories with hashed names, which makes them impossible to manage effectively, especially with regard to minimal filesystem size footprint.
|
||||
|
||||
Docker requires a daemon to run at all times. Whenever the daemon needs to be stopped or restarted (e.g. due to an update), all containers need to be stopped too. It is also yet another a single point of failure. LXC is daemonless, as it simply prepares the individual container namespaces and launches the init process in them.
|
||||
|
||||
Finally, Docker maintainers explicitly refuse to implement a feature which would allow to restrict the Docker daemon to private repositories (registries) in the community edition of Docker (*Docker Enterprise* allows this). It is possible to have custom and even private repositories, but it is not possible to deactivate the default public *Dockerhub*.
|
||||
|
||||
The downside of using LXC is that its usage requires more knowledge about how the linux containers actually work. Another problem is fast availability of the desired image as most 3rd party applications are shipped with `Dockerfile` or directly distributed as Docker images. SPOC/LXC requires rewriting into LXC-compatible containers, however this is simplified by SPOC as the `image building <spoc-builder>`_ aims to mimic the features of Docker and automatize LXC container building using *Dockerfile*-like syntax.
|
||||
|
||||
In the future, LXC and part of SPOC may be replaced by Podman and Podman-compose to reach even wider audience.
|
||||
|
||||
Package manager
|
||||
---------------
|
||||
|
||||
SPOC uses custom simplistic built-in package manager. There are several benefits of using custom package management which outweigh the shortcomings of native APK package manager.
|
||||
|
||||
Native packaging toolchain (abuild) is designed for automated bulk package building. Building packages from pre-existing directories requires some customizations and workarounds to strip unnecessary (resp. outright harmful) steps like binary stripping, symlink resolution and dependency tracing. It also requires to be run under non-root user inside ``fakeroot`` which is problematic when LXC containers should be packaged. Most of the limitations can be worked around (run as root using ``-F``, spoof build process by bind-mounting existing directory to packaging directory, skip dependency tracing using ``options="!tracedeps"`` in ``APKFILE`` and omit majority of the build process by running only ``build package prepare_metafiles create_apks index clean`` abuild targets), however there is no real benefit in abusing the native tools this way.
|
||||
|
||||
Furthermore, when ``apk`` package manager installs a package, it first unpacks it, then runs post-install script and once all packages are installed, only then it modifies the permissions and ownership of the files to the original values contained in the package. This means that it's not possible to run container setup as part of post-install script as most applications require the permissions to be already correct. Every single file including its ownership and permissions along with a hash is recorded in ``/lib/apk/db/installed``, which only unnecessarily bloats the database of locally installed packages (e.g. the basic python 3 layer contains ~6500 files).
|
||||
|
||||
With custom package manager, the whole download, unpacking and installation process can be observed directly, keeping the user informed about the currently ongoing step, as opposed to a mere download percentage offered by the bare ``apk``. Finally, the APK packages are only gzipped whereas the custom solution uses xz (LZMA2), allowing for up to 70% smaller packages, which is one of the SPOC goals.
|
||||
|
||||
SPOC recognizes three types of repositories, all of which work in the same fashion (i.e. with ``.tar.xz`` archives and JSON metadata)
|
||||
|
||||
- Local repository - Contains information about all layers, images, containers and applications installed and available on the local machine.
|
||||
- Online repository - A remote repository containing images and applications available for download and installation. Unlike traditional package manages, SPOC can have configured only a single online repository.
|
||||
- Publish repository - Local directory where the builder stages archives and metadata, which are later supposed to be copied to the online repository. SPOC doesn't handle the copying, so that remains at discretion of the builder.
|
||||
|
||||
Deliverables
|
||||
------------
|
||||
|
||||
Layers
|
||||
^^^^^^
|
||||
|
||||
A layer is a collection of files which logically belong together and form a product or a group of products, which can be reused in a larger unit.
|
||||
|
||||
The term *layer* is used because the storage is handled by OverlayFS filesystem consisting of groups of files overlaid over each other. This allows us to have e.g. a layer with the basic operating system, another layer with runtime, and final layer with the application itself. This reduces the necessity to duplicate functionality in every container and waste disk space.
|
||||
|
||||
Images
|
||||
^^^^^^
|
||||
|
||||
An image is a layer or a collection of layers with attached metadata describing how to work with the layers. It is basically a blueprint for container creation.
|
||||
|
||||
The metadata describe the order of OverlayFS layers, init working directory, user and command to be executed on startup etc. Images are closely tied to layers, therefore they always have the same name as their topmost layer. SPOC doesn't allow to create layers without images, so even with no metadata, it will be possible to spawn a container out of a layer (resp. image with no metadata) if all parent layers exist. Metadata from parent image are always overridden by metadata of the image which builds on top of it.
|
||||
|
||||
Each layer is packaged as a separate installable archive and the associated image's metadata are written in the repository manifest.
|
||||
|
||||
Containers
|
||||
^^^^^^^^^^
|
||||
|
||||
A singular runnable instance of an image with additional metadata specific for the purpose of the container instance - e.g. persistent volume paths.
|
||||
|
||||
All user-installable applications run in containers. A *container* is defined by following settings combined from the image from which it has been created and additional metadata:
|
||||
|
||||
- Network type and interface configuration
|
||||
- OverlayFS storage layers
|
||||
- Mountpoints to store persistent data
|
||||
- Functional user and binary to be executed on startup
|
||||
- Environment variables propagated to the container namespace
|
||||
- Signal used to stop the container
|
||||
- TTY / console logging
|
||||
- Syscall capability restrictions
|
||||
- Event hooks (these are used by SPOC to circumvent some shortcomings of LXC, resp. OverlayFS)
|
||||
|
||||
Applications
|
||||
^^^^^^^^^^^^
|
||||
|
||||
A collection of distinct containers working together to form an environment dedicated to the given application. Apart from metadata, application package includes also scripts for installation, update and uninstallation of such environment on a host system.
|
||||
|
||||
The application should be the final product exposed to the user, which the user can effortlessly install and run, without thinking about integration with other components, persistence or migrations between versions during updates.
|
||||
|
||||
Example
|
||||
^^^^^^^
|
||||
|
||||
To illustrate the types, let's take a simple web application - e.g. WordPress. WordPress requires a web server, PHP interpreter and a database to function. The aim of the layers and images is to be reusable in multiple applications. We can logically split the WordPress image into several lower layers/images.
|
||||
|
||||
- Basic OS layer - Can be later reused for any other images which we wish to build on the same OS.
|
||||
- nginx+PHP layer - On top of the OS layer, we create another layer with nginx web server and PHP interpreter, as these can be reused again for any other web application requiring the same components.
|
||||
- WordPress layer - Finally, on top of the nginx+PHP layer we create a layer with the actual WordPress files.
|
||||
- MySQL layer - Since we need also a database and it's likely that another application will also need a database, we base our MySQL layer on the basic OS layer again. That way we deduplicate the basic OS files and reduce the amount of data user needs to download and store.
|
||||
|
||||
Once the images are build, we create the application definition, which instructs SPOC that two containers should be spawned. One from the image with WordPress and the other from the image with MySQL. We also need to write a short installation script to set up the application environment - e.g. populate the database and create an admin user.
|
||||
|
||||
Finally we publish the images and the application and end up with following structure:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Basic OS layer/image
|
||||
├─ nginx+PHP layer/image
|
||||
│ └─ WordPress layer/image
|
||||
└─ MySQL layer/image
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
WordPress application definition
|
||||
├─ WordPress container definition based on WordPress image
|
||||
├─ MySQL container definition based on MySQL image
|
||||
└─ install.sh script
|
||||
|
||||
The end-user then issues a single command to install the WordPress application which will download all the required layers, register the image metadata, create the containers, execute the installation script and finally register the application metadata too.
|
||||
275
doc/toolchain/spoc-user.rst
Normal file
275
doc/toolchain/spoc-user.rst
Normal file
@ -0,0 +1,275 @@
|
||||
SPOC user manual
|
||||
==================
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
In order to consume the images and applications from online repositories, the user needs to set up the online repository URL and ECDSA public key in their ``/etc/spoc/spoc.conf`` file. The configuration file directives are covered on the `architecture <spoc-architecture>`_ page.
|
||||
|
||||
Images
|
||||
------
|
||||
|
||||
List images
|
||||
^^^^^^^^^^^
|
||||
|
||||
The user can choose to download any image and spawn a separate standalone container, not tied to any application. To list the images in the respective repositories, ``spoc-image list`` command is used.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image list [{installed,online,published}]
|
||||
|
||||
positional arguments:
|
||||
{installed,online,published}
|
||||
Selected repository
|
||||
|
||||
Download image
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
Then the user needs to issue ``spoc-image download`` to download the image and make it available locally.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image download image
|
||||
|
||||
positional arguments:
|
||||
image Name of the image to download
|
||||
|
||||
Delete image
|
||||
^^^^^^^^^^^^
|
||||
|
||||
If the user wishes to delete locally installed image, they need to run ``spoc-image delete``.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image delete image
|
||||
|
||||
positional arguments:
|
||||
image Name of the image to delete
|
||||
|
||||
Cleanup unused images
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If the user wants to clean up all locally installed images which are not used by any container, there is ``spoc-image clean`` to do just that.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-image clean
|
||||
|
||||
Containers
|
||||
----------
|
||||
|
||||
Create container
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
To create a container based on locally installed image, the user needs to use ``spoc-container create`` command. This command has a lot of parameters, making it possible to customize every aspect of the container supported by SPOC.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container create [-d DEPENDS] [-m MOUNT] [-e ENV] [-u UID] [-g GID] [-c CMD] [-w WORKDIR] [-r READY] [-s STOPSIG] container image
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to create
|
||||
image Name of the image of which the container should be based
|
||||
|
||||
optional arguments:
|
||||
-d DEPENDS, --depends DEPENDS
|
||||
Add another container as a start dependency
|
||||
-m MOUNT, --mount MOUNT
|
||||
Add mount to the container - format volume:mountpoint[:file]
|
||||
-e ENV, --env ENV Add environment variable for the container - format KEY=value
|
||||
-u UID, --uid UID Sets the container init UID
|
||||
-g GID, --gid GID Sets the container init GID
|
||||
-c CMD, --cmd CMD Sets the container init command
|
||||
-w WORKDIR, --workdir WORKDIR
|
||||
Sets the container init working directory
|
||||
-r READY, --ready READY
|
||||
Sets the container ready command
|
||||
-s STOPSIG, --stopsig STOPSIG
|
||||
Sets the signal to be sent to init on container shutdown
|
||||
|
||||
Modify container
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
In similar fashion, ``spoc-container modify`` can be used to update the configuration of existing container.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container modify [-d DEPENDS] [-m MOUNT] [-e ENV] [-u UID] [-g GID] [-c CMD] [-w WORKDIR] [-r READY] [-s STOPSIG] container
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to modify
|
||||
|
||||
optional arguments:
|
||||
-d DEPENDS, --depends DEPENDS
|
||||
Add another container as a start dependency - prepend the name with ! to remove the dependency
|
||||
-m MOUNT, --mount MOUNT
|
||||
Add mount to the container - format volume:mountpoint - specify empty mountpoint to remove the mount
|
||||
-e ENV, --env ENV Add environment variable for the container - format KEY=value - specify empty value to remove the env
|
||||
-u UID, --uid UID Sets the container init UID
|
||||
-g GID, --gid GID Sets the container init GID
|
||||
-c CMD, --cmd CMD Sets the container init command
|
||||
-w WORKDIR, --workdir WORKDIR
|
||||
Sets the container init working directory
|
||||
-r READY, --ready READY
|
||||
Sets the container ready command
|
||||
-s STOPSIG, --stopsig STOPSIG
|
||||
Sets the signal to be sent to init on container shutdown
|
||||
|
||||
Destroy container
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
To destroy the container, i.e. to unregister from metadata and remove its LXC configuration, ``spoc-container destroy`` should be used.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container destroy container
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to destroy
|
||||
|
||||
The containers can be started and stopped via commands ``spoc-container start`` and ``spoc-container stop``. The start command allows to override the default init command.
|
||||
|
||||
Start / stop container
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container start container ...
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to start
|
||||
command Command to be run instead of the default init command
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container stop container
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to stop
|
||||
|
||||
Check container status
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to check status of a container ``spoc-container status`` command can be issued.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container status [-h] container
|
||||
|
||||
positional arguments:
|
||||
container Name of the container to check
|
||||
|
||||
Execute command in a container
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Finally, whenever a command needs to be run in context of the container, ``spoc-container exec`` can be used. This command automatically determines if the container is stopped or started and depending on that, it either created the namespace or just attach to the already existing one.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-container exec [-u UID] [-g GID] container ...
|
||||
|
||||
positional arguments:
|
||||
container Name of the container in which to run the command
|
||||
command The command to be run
|
||||
|
||||
optional arguments:
|
||||
-u UID, --uid UID Sets the command UID
|
||||
-g GID, --gid GID Sets the command GID
|
||||
|
||||
Applications
|
||||
------------
|
||||
|
||||
List applications
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
In similar fashion like the images, the user can list also the applications and filter them by repository or status. This is done via ``spoc-app list`` command.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app list [{installed,online,updates,published,running,stopped}]
|
||||
|
||||
positional arguments:
|
||||
{installed,online,updates,published,running,stopped}
|
||||
Selected repository or application criteria
|
||||
|
||||
Install application
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To install an application, the user issues ``spoc-app install`` command. SPOC's dependency solver and task queue ensures that the components used by the application are made available first before the application is attempted to be installed.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app install app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to install
|
||||
|
||||
Update application
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In the same way as installation, the user can update locally installed application via ``spoc-app update``.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app update app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to update
|
||||
|
||||
Uninstall application
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To uninstall the application, the user runs ``spoc-app uninstall``. Note that this command only uninstalls the application and removes its containers, leaving all the images formerly used by them locally registered and installed. In order to remove them for good, user may execute ``spoc-image clean`` described above.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app uninstall app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to uninstall
|
||||
|
||||
Start / stop application
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to start or stop the application, resp. all its containers in the order given by the dependencies in metadata, the user can issue ``spoc-app start`` and ``spoc-app stop`` respectively.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app start app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to start
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app stop app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to stop
|
||||
|
||||
|
||||
Check application status
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To check the application status, the user can use ``spoc-app status``. The application is just a group of containers, so the output will simply display status of all containers in the given application environment.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app status app
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to check
|
||||
|
||||
Set application autostart
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To have the application automatically started after the VM boots up (resp. when the host's init executes the SPOC service), one can use ``spoc-app autostart``. This option exists only for applications, not for the individual containers.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: spoc-app autostart app {1,on,enable,true,0,off,disable,false}
|
||||
|
||||
positional arguments:
|
||||
app Name of the application to be automatically started
|
||||
{1,on,enable,true,0,off,disable,false}
|
||||
Set or unset the applications to be automatically started after the host boots up
|
||||
73
doc/toolchain/virtual-machine-creation.rst
Normal file
73
doc/toolchain/virtual-machine-creation.rst
Normal file
@ -0,0 +1,73 @@
|
||||
Virtual machine creation
|
||||
========================
|
||||
|
||||
Virtual machine specifications
|
||||
------------------------------
|
||||
|
||||
- **Memory:** 4 GB
|
||||
- **CPU:** 1 processor, 2 cores
|
||||
- **Hard Disk:** SCSI, 300 MB
|
||||
- **CD/DVD**: IDE
|
||||
- **Network Adapter**: Bridged
|
||||
|
||||
In case you're setting up a VMWare virtual machine, select OS type *Other Linux 3.x kernel 64-bit* and after the VM is created, manually edit the ``*.vmx`` file using a text editor and add ``mem.hotadd = "FALSE"``. Failing to do so will result in system unable to boot. Other hypervisors don't need this adjustment.
|
||||
|
||||
Virtual machine installation
|
||||
----------------------------
|
||||
|
||||
Note that Alpine is getting new versions fairly quickly. At the time of writing this documentation, 3.11.6 was the latest version, however it's expected that the VM will be kept up-to-date and the instructions written in this document will work even for future versions.
|
||||
|
||||
Download **Alpine Virtual x86_64** from https://alpinelinux.org/downloads/ and boot from it. At the login prompt, use the root user without password to log in.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Set up interfaces (leave the default choices)
|
||||
setup-interfaces
|
||||
ifup eth0
|
||||
|
||||
# Download and launch the setup script
|
||||
wget https://repo.spotter.cz/vm.sh
|
||||
sh vm.sh
|
||||
|
||||
The script will perform installation and configuration of Alpine linux, LXC and the whole VMMgr platform. Virtual machine is protected by LUKS-on-LVM disk encryption. The encryption password, which is simultaneously also a password for VMMgr web administration interface, will be asked for at the beginning of the script execution. Root access on TTY is disabled.
|
||||
|
||||
After the script finishes and shuts down the virtual machine, remove CD/DVD drive from the virtual machine settings and extend the HDD to 80 GB. This is to minimize the overall size of the OVA (Open Virtual Appliance) file to which the VM will be exported.
|
||||
|
||||
Build environment installation
|
||||
------------------------------
|
||||
|
||||
Follow the VM creation steps as above. Once the VM is built and restarted, log in to the web interface and enter your SSH public key into *authorized_keys* from on ``/setup-apps``. Then log in via SSH.
|
||||
|
||||
First time setup
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Install git and OpenSSH client
|
||||
apk add git openssh-client
|
||||
|
||||
# Create SSH key
|
||||
ssh-keygen -t ecdsa
|
||||
|
||||
Assign the newly generated key to your GitLab account
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Clone the repository
|
||||
git clone --recursive ssh://git@git.spotter.cz:2222/Spotter-Cluster/Spotter-Cluster.git
|
||||
|
||||
# Install the build toolchain
|
||||
Spotter-Cluster/build/install-toolchain.sh
|
||||
|
||||
Building the packages
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
There are 3 distinct packaging systems.
|
||||
|
||||
1. Just a plain *tar.gz* for basic OS setup used by ``vm.sh`` installation script.
|
||||
2. `Abuild <abuild>`_ for the native Alpine linux packages (APK) used for ACME client and VMMgr packaging.
|
||||
3. `SPOC <spoc>`_ for SPOC container building and packaging.
|
||||
|
||||
Before any building and packaging can be started, build toolchain including signing keys needs to be set up. This is done via ``install-toolchain.sh`` script.
|
||||
|
||||
One the setup is complete, build and packaging can be done manually on per-container basis or a full build of all components can be run via ``build-all.sh`` script. The full build process takes considerable amount of time, so it is advised to have a persistent build VM and rebuild only updated packages.
|
||||
45
doc/toolchain/virtual-machine-internals.rst
Normal file
45
doc/toolchain/virtual-machine-internals.rst
Normal file
@ -0,0 +1,45 @@
|
||||
Virtual machine internals
|
||||
=========================
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The virtual machine runs Alpine linux installed with LVM on LUKS disk encryption. The installation steps are loosely based on the official walkthrough on `LVM on LUKS <https://wiki.alpinelinux.org/wiki/LVM_on_LUKS>`_ Alpine wiki page.
|
||||
|
||||
Alpine linux is minimalistic linux distribution using *musl c* library (as opposed to *glibc*) and OpenRC init system. By default, the distro aims to be strictly POSIX-compliant. The main set of utilities is based on ``busybox``, however the full-featured variants of any well-known linux utilities can be installed. Installation of any package is done via APK (*Alpine Package Keeper*) which is the Alpine-native package manager. Package repository can be browsed on `pkgs.alpinelinux.org <https://pkgs.alpinelinux.org/packages>`_.
|
||||
|
||||
Disk partitioning
|
||||
-----------------
|
||||
|
||||
By default, the VM is created with 300 MB hard drive divided into two partitions. The first partition is 50 MB in size and serves as ``/boot`` partition for *EXTLINUX* bootloader. The second partition with 250 MB in size serves as physical volume for encryption. The encrypted LUKS volume spans over the whole partition and is called ``system``. This volume is part of ``vg0`` LVM volume group. On this volume group, logical volume named ``root`` formatted as ext4 is created spanning over the whole ~250 MB and mounted as ``/``.
|
||||
|
||||
Such small initial size of the VM is needed to minimize the size of virtual appliance export as it forces the ext4 to cram all it needs to write to that 250 MB range and overwrite freed space if needed. Once the initial installation is done, the VM disk can be extended to any larger size. Somewhere between 40 and 80 GB is plenty for the start. VM is then immediately exported as virtual appliance (OVA) without any subsequent runs.
|
||||
|
||||
There is a script ``/sbin/extend-disk`` which is called by ``vmtty`` (see Pseudoterminal_) which compares the total size of the virtual disk device with size allocated by the filesystems. If the size differs and the device is more that 10 MB larger, the script triggers adjustment on all filesystem levels (i.e. LUKS volume, LVM physical volume, LVM logical volume and the ext4 filesystem). A password for LUKS encryption is requested to be input by the user. In case the virtual disk is larger by 4 GB or more and the swap volume doesn't exist yet, the script automatically creates it. All the steps are expected to happen at least once, on the first run when the user imports and starts the virtual appliance.
|
||||
|
||||
The final filesystem layout is expected to look as follows:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
spotter:~# lsblk -o name,fstype,size,mountpoint
|
||||
NAME FSTYPE SIZE MOUNTPOINT
|
||||
sda 80G
|
||||
├─sda1 ext4 50M /boot
|
||||
└─sda2 crypto_LUKS 80G
|
||||
└─system LVM2_member 80G
|
||||
├─vg0-root ext4 76G /
|
||||
└─vg0-swap swap 4G
|
||||
|
||||
Pseudoterminal
|
||||
--------------
|
||||
|
||||
There is a customized terminal executable ``/sbin/vmtty``, which is spawned and handled via ``getty`` on VM startup. The purpose of the terminal is to execute the ``extend-disk`` script described above and display contents of ``/etc/issue``. The terminal does not allow login and servers solely for informative purposes. The content of ``/etc/issue`` is initially the Alpine default, but it is populated by VMMgr via ``vmmgr rebuild-issue`` every time the terminal process is spawned.
|
||||
|
||||
Traditionally, there are several pseudoterminals spawned on linux machines, including one listening on serial port. SpotterVM disables all terminals. All interaction is expected to be done via VMMgr, however VMMgr allows to add SSH key for root access, so if the need arises, the root console can be obtained.
|
||||
|
||||
Since the primary usage is expected to be in Central Europe, the terminal is expected to support fonts with diacritics. This is achieved by reconfiguration of ``consolefont`` to specifically support ``ISO 8859-2`` and enabling unicode support in OpenRC.
|
||||
|
||||
Web server
|
||||
----------
|
||||
|
||||
By default, the only access to any VM features is available only through VMMgr, which runs as a standalone WSGI application. All access passes through *nginx* web server which serves as reverse proxy. See `VMMgr internals <vmmgr-internals>`_ for more detail on the nginx configuration.
|
||||
@ -1,65 +0,0 @@
|
||||
# Virtual machine creation
|
||||
|
||||
## Virtual machine specifications
|
||||
|
||||
- **Memory:** 4 GB
|
||||
- **CPU:** 1 processor, 2 cores
|
||||
- **Hard Disk:** SCSI, 300 MB
|
||||
- **CD/DVD**: IDE
|
||||
- **Network Adapter**: Bridged
|
||||
|
||||
In case you're setting up a VMWare virtual machine, select OS type *Other Linux 3.x kernel 64-bit* and after the VM is created, manually edit the `*.vmx` file using a text editor and add `mem.hotadd = "FALSE"`. Failing to do so will result in system unable to boot. Other hypervisors don't need this adjustment.
|
||||
|
||||
## Virtual machine installation
|
||||
|
||||
Download **Alpine Virtual 3.9.0 x86_64** from <https://alpinelinux.org/downloads/> and boot from it. At the login prompt, use the root user without password to log in.
|
||||
|
||||
```bash
|
||||
# Set up interfaces (leave the default choices)
|
||||
setup-interfaces
|
||||
ifup eth0
|
||||
|
||||
# Download and launch the setup script
|
||||
wget https://repo.spotter.cz/vm.sh
|
||||
sh vm.sh
|
||||
```
|
||||
|
||||
The script will perform installation and configuration of Alpine linux, LXC and the whole VMMgr platform. Virtual machine is protected by LUKS-on-LVM disk encryption. The encryption password, which is simultaneously also a password for VMMgr web administration interface, will be asked for at the beginning of the script execution. Root access is disabled.
|
||||
|
||||
After the script finishes and shuts down the virtual machine, remove CD/DVD drive from the virtual machine settings and extend the HDD to 80 GB. This is to minimize the overall size of the OVA (Open Virtual Appliance) file to which the VM will be exported.
|
||||
|
||||
## Build environment installation
|
||||
|
||||
Follow the VM creation steps as above. Once the VM is built and restarted, log in to the web interface and enter your SSH public key into *authorized_keys* from on `/setup-apps`. Then log in via SSH.
|
||||
|
||||
### First time setup
|
||||
|
||||
```bash
|
||||
# Install git and OpenSSH client
|
||||
apk add git openssh-client
|
||||
|
||||
# Create SSH key
|
||||
ssh-keygen -t ecdsa
|
||||
```
|
||||
|
||||
Assign the newly generated key to your GitLab account
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone --recursive ssh://git@git.spotter.cz:2222/Spotter-Cluster/Spotter-Cluster.git
|
||||
|
||||
# Install the build toolchain
|
||||
Spotter-Cluster/build/install-toolchain.sh
|
||||
```
|
||||
|
||||
### Building the packages
|
||||
|
||||
There are 3 distinct packaging systems.
|
||||
|
||||
1. Just a plain tar for basic OS setup used by `vm.sh` installation script.
|
||||
2. [Abuild](abuild) for the native Alpine linux packages (APK) used for ACME client and VMMgr packaging.
|
||||
3. [`lxcbuild`](lxcbuild) / [`lxc-pack`](lxc-pack) for LXC container building and packaging.
|
||||
|
||||
Before any building and packaging can be started, build toolchain including signing keys needs to be set up. This is done via `install-toolchain.sh` script.
|
||||
|
||||
One the setup is complete, build and packaging can be done manually on per-container basis or a full build of all components can be run via `build-all.sh` script. The full build process takes considerable amount of time, so it is advised to have a persistent build VM and rebuild only updated packages.
|
||||
@ -1,90 +0,0 @@
|
||||
# VMMgr command line hooks
|
||||
|
||||
VMMgr is mostly a WSGI web GUI application, but there are few specific cases when it is invoked via command line.
|
||||
|
||||
## Installation hooks
|
||||
|
||||
### register-app
|
||||
|
||||
This hook is invoked by [Package manager](pkgmgr), resp. `install.sh` script of the respective installing package. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr register-app <application> <subdomain> [admin-username] [admin-password]
|
||||
```
|
||||
|
||||
Where the `application` is the internal application name, same as previously used in the package name, `subdomain` is the subdomain on which the application will be accessible and the `admin-username` and `admin-password` are optional admin credentials created by `install.sh` during the application setup. These parameters will show up as part of the application definition in `/etc/vmmgr/conf.json` and will be visible / usable via web GUI.
|
||||
|
||||
### unregister-app
|
||||
|
||||
Counterpart to `register-app`. This hook is invoked by `uninstall.sh` and simply removes the application definition from `/etc/vmmgr/conf.json`, rendering it invisible / unusable via web GUI, which is expected as part of the uninstallation process. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr unregister-app <application>
|
||||
```
|
||||
|
||||
Where the `application` is the internal application name, same as previously used in the package name.
|
||||
|
||||
## LXC hooks
|
||||
|
||||
LXC hooks set various environment variables prior to calling the defined executables. For overview of native LXC hooks, see section *Container hooks* in the official [lxc.container.conf(5) documentation](https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html). All hooks mentioned in this chapter are hardcoded in the container configuration via a template used by[`lxcbuild`](lxcbuild).
|
||||
|
||||
### prepare-container
|
||||
|
||||
This hook is invoked by LXC's `lxc.hook.pre-start`, which is called in host environment (i.e. in the virtual machine, not in the container) before the container's ttys, consoles or mounts are up. This hook removes the contents of container's ephemeral OverlayFS layer in case of previous unclean container shutdown. Then calls application's `update-conf.sh` script (if it is present) with environment variables populated from configuration in `/etc/vmmgr/conf.json`, allowing for the correct configuration of application in components which are not part of the currently starting container, e.g. application URL in a database or persistent configuration file. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr lxc-container
|
||||
```
|
||||
|
||||
The hook expects the `LXC_NAME` environment variable to be populated.
|
||||
|
||||
### register-container
|
||||
|
||||
This hook is invoked by LXC's `lxc.hook.start-host` which is called in host environment (i.e. in the virtual machine, not in the container) after the container has been setup, immediately before launching the container's init process. This hook sets up a container's network settings (IP address, netmask, default route). The network settings is managed via *poor man's DHCP server*, which simply assigns the first unassigned IP address from range 172.17.0.0/16 and holds the lease table in hosts's `/etc/hosts` which is also mounted in all containers, keeping the network settings consistent across all container stop / starts. The hook is invoked as
|
||||
|
||||
```
|
||||
vmmgr register-container
|
||||
```
|
||||
|
||||
The hook expects `LXC_NAME` and `LXC_PID` environment variables to be populated, so it can record the value of `LXC_NAME` into `/etc/hosts` leases and then `nsenter` the container namespace via `LXC_PID` and set the leased address.
|
||||
|
||||
|
||||
### unregister-container
|
||||
|
||||
This hook is invoked by LXC's `lxc.hook.post-stop` which is called in host environment (i.e. in the virtual machine, not in the container) after the container has been shut down. This hook releases the IP address leased by `register-container` and then removes the contents of container's ephemeral OverlayFS layer. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr unregister-container
|
||||
```
|
||||
|
||||
The hook expects the `LXC_NAME` environment variable to be populated.
|
||||
|
||||
## Init hooks
|
||||
|
||||
### rebuild-issue
|
||||
|
||||
This hook is called by `/sbin/vmtty`, which is the default *login program* used by `/sbin/getty` defined in `/etc/inittab`. This tty contains a banner with branding, legal notice and information about VM network settings and URL, stored in `/etc/issue`. The URL needs to be generated based on the settings in `/etc/vmmgr/conf.json`, which is exactly what this hook does. On every invocation, it regenerates the contents of `/etc/issue` with up-to-date information. The hook is invoked as-is with no parameters or environment variables.
|
||||
|
||||
```
|
||||
vmmgr rebuild-issue
|
||||
```
|
||||
|
||||
### register-proxy
|
||||
|
||||
This hook is invoked by `/etc/init.d/<application>` OpenRC init script as part of `start_post()` function. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr register-proxy <application>
|
||||
```
|
||||
|
||||
The hook creates HTTP proxy configuration file for VM's nginx HTTP server for the given `application`, based on the settings in `/etc/vmmgr/conf.json`. Then it reloads the nginx service to take the configuration into effect. The proxy configuration requires a valid IP/hostname in `/etc/hosts`, which exists only after the container has been started and `register-container` hook called, hence the post-start invocation.
|
||||
|
||||
### unregister-proxy
|
||||
|
||||
Counterpart to `register-proxy`. This hook is called by `/etc/init.d/<application>` OpenRC init script as part of `stop_pre()` function. It is invoked as
|
||||
|
||||
```
|
||||
vmmgr unregister-proxy <application>
|
||||
```
|
||||
|
||||
The hook simply removes the configuration file for VM's nginx HTTP server for the given `application` and reloads the nginx service to take the configuration out of effect.
|
||||
207
doc/toolchain/vmmgr-internals.rst
Normal file
207
doc/toolchain/vmmgr-internals.rst
Normal file
@ -0,0 +1,207 @@
|
||||
VMMgr internals
|
||||
===============
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
While VMMgr mainly utilizes metadata and configuration of SPOC and other tools, it has its own configuration file with metadata which are either sourced to other tools or are relevant only in context of VMMgr, resp. its web interface and nginx reverse proxy. The JSON configuration file is located in ``/etc/vmmgr/config.json`` and its structure looks as follows
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"apps": {
|
||||
"sahana": {
|
||||
"host": "sahana",
|
||||
"login": "admin@example.com",
|
||||
"password": "KlLwlo3DxW3sK7gW",
|
||||
"visible": true
|
||||
}
|
||||
},
|
||||
"common": {
|
||||
"email": "admin@example.com",
|
||||
"gmaps-api-key": ""
|
||||
},
|
||||
"host": {
|
||||
"adminpwd": "$2b$12$1QAv6NEuHCGWbP8IqjhZ/ehxMbW1jwcUBptYgzg1CVmro9FBrQfPO",
|
||||
"domain": "spotter.vm",
|
||||
"port": "8443"
|
||||
}
|
||||
}
|
||||
|
||||
host
|
||||
^^^^
|
||||
|
||||
The ``host`` part in ``domain`` and ``port`` fields holds easily parsable information about the main HTTP host on which the VMMgr is accessible. This configuration is supplied to other services and tools such as nginx and `Issue / MotD`_ banners. The HTTP host component is also used in conjunction with the application subdomain to form the FQDN for the separate applications which the nginx reverse proxy accepts.
|
||||
|
||||
The ``adminpwd`` part is *argon2* hash of the VMMgr administrator password which is designed to be the same as the LUKS disk encryption password (see `Virtual Machine internals <virtual-machine-internals>`_). However, there isn't any direct link between these two and VMMgr always attempts to modify both passwords at once, leveraging the return code of ``cryptsetup luksChangeKey`` to obtain the information about whether the user managed to supply the correct old password.
|
||||
|
||||
common
|
||||
^^^^^^
|
||||
|
||||
The ``common`` part currently contains settings which is important in the context of some applications, such as SMTP sender email or Google API keys. These settings are propagated to the applications via ``update-conf.sh`` script located in the respective SPOC application directories. However this system is not extensible and needs to be reworked on per-application settings as described in `VMMgr issue #4 <https://git.spotter.cz/Spotter-Cluster/vmmgr/-/issues/4>`_.
|
||||
|
||||
apps
|
||||
^^^^
|
||||
|
||||
The ``apps`` part contains metadata for applications. In the example above, there is a record for ``sahana`` SPOC application with nginx proxy host defined as ``sahana``, creating the full HTTP host as ``sahana.spotter.vm:8443``.
|
||||
|
||||
The ``login`` and ``password`` fields are plaintext username and password for the application which were automatically generated during the application setup. This is only to display the generated password to the VMMgr administrator. The username and password aren't connected to the actual application in any way, so when the user modifies username or password directly in the application, it is not reflected in VMMgr configuration.
|
||||
|
||||
The ``visible`` field stores information to determine if the application should be displayed on VMMgr's application portal. In order for the application to be visible, it needs to be both stated and have this filed set to ``true``. However, setting this field to ``false`` does not prevent HTTP requests to be routed by nginx reverse proxy and processed by the application. The setting is purely cosmetical.
|
||||
|
||||
Nginx configuration
|
||||
-------------------
|
||||
|
||||
VMMgr runs as a standalone WSGI application. All HTTP requests are passed through *nginx* HTTP server which serves as reverse proxy. The web server processes all HTTP and HTTPS connections for VMMgr and SPOC applications and containers. VMMgr is the component setting up the proxy rules based on a common template.
|
||||
|
||||
The web server is configured to redirect HTTP to HTTPS. On HTTPS, both TLSv1.2 and TLSv1.3 are supported, as some applications are making callbacks which are passing through nginx and are too old and can't handle TLS 1.3 handshakes. The rest of the TLS settings generally tries to follow `Mozilla Guidelines <https://ssl-config.mozilla.org/#server=nginx&config=modern&hsts=false&ocsp=false>`_ for modern browsers where possible.
|
||||
|
||||
The default nginx configuration relevant to VMMgr looks as follows
|
||||
|
||||
.. code-block:: nginx
|
||||
|
||||
server {
|
||||
listen [::]:80 default_server ipv6only=off;
|
||||
|
||||
location / {
|
||||
return 301 https://$host:443$request_uri;
|
||||
}
|
||||
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /etc/acme.sh.d;
|
||||
}
|
||||
|
||||
location = /vm-ping {
|
||||
add_header Content-Type text/plain;
|
||||
return 200 "vm-pong";
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen [::]:443 ssl http2 default_server ipv6only=off;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
}
|
||||
|
||||
location /static {
|
||||
root /usr/share/vmmgr;
|
||||
}
|
||||
|
||||
error_page 502 /502.html;
|
||||
location = /502.html {
|
||||
root /usr/share/vmmgr/templates;
|
||||
}
|
||||
|
||||
location = /vm-ping {
|
||||
add_header Content-Type text/plain;
|
||||
return 200 "vm-pong";
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen [::]:443 ssl http2;
|
||||
server_name *.spotter.vm;
|
||||
|
||||
location / {
|
||||
proxy_pass http://172.17.0.2:8080;
|
||||
}
|
||||
}
|
||||
|
||||
The template for individual applications looks as
|
||||
|
||||
.. code-block:: nginx
|
||||
|
||||
server {
|
||||
listen [::]:443 ssl http2;
|
||||
server_name sahana.spotter.vm;
|
||||
|
||||
access_log /var/log/nginx/sahana.access.log;
|
||||
error_log /var/log/nginx/sahana.error.log;
|
||||
|
||||
location / {
|
||||
proxy_pass http://172.17.0.2:8080;
|
||||
}
|
||||
|
||||
include vmmgr_common;
|
||||
}
|
||||
|
||||
Where the application name is taken from metadata and the upstream IP from SPOC global hosts file (see `Networking chapter in SPOC Architecture <spoc-architecture#networking>`_).
|
||||
|
||||
The ``vmmgr_common`` is a file with several static rules
|
||||
|
||||
.. code-block:: nginx
|
||||
|
||||
error_page 502 /502.html;
|
||||
location = /502.html {
|
||||
root /usr/share/vmmgr/templates;
|
||||
}
|
||||
|
||||
error_page 503 /503.html;
|
||||
location = /503.html {
|
||||
root /usr/share/vmmgr/templates;
|
||||
}
|
||||
|
||||
location = /vm-ping {
|
||||
add_header Content-Type text/plain;
|
||||
return 200 "vm-pong";
|
||||
}
|
||||
|
||||
The 502 and 503 error pages are briefly describing the state of the application and they are usually displayed when the application is not fully started. The ``/vm-ping`` endpoint is used for checks if the applications are available from the internet on their respective subdomains.
|
||||
|
||||
Application hooks
|
||||
-----------------
|
||||
|
||||
VMMgr needs to be explicitly told whenever there is a new SPOC application installed to register its metadata and nginx reverse proxy. This is achieved via ``vmmgr register-app`` and ``vmmgr unregister-app`` calls respectively.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: vmmgr register-app app host [login] [password]
|
||||
|
||||
positional arguments:
|
||||
app Application name
|
||||
host Application subdomain
|
||||
login Admin login
|
||||
password Admin password
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
usage: vmmgr unregister-app app
|
||||
|
||||
positional arguments:
|
||||
app Application name
|
||||
|
||||
The registration add the passed parameters as metadata to the `configuration`_. At the same time it also creates the nginx reverse proxy configuration file described above using a template.
|
||||
|
||||
Removal of orphaned images
|
||||
--------------------------
|
||||
|
||||
SPOC normally doesn't remove unused and orphaned images / layers whenever an application is uninstalled. Unlike SPOC, VMMgr is intended to be used more naively, so unlike SPOC, VMMgr *does* remove all unused and orphaned images / layers whenver an application is uninstalled. This may lead to confusing scenarios when both SPOC commands and VMMgr is used on the same machine, as uninstallation or update of one application via SPOC doesn't remove its orphaned layers, but subsequent uninstallation / update of another application via VMMgr will remove layers for both the first and the second application, possibly leading the user to believe that there is any connection between the two applications when there is none.
|
||||
|
||||
Issue / MotD
|
||||
------------
|
||||
|
||||
VMMgr handles generation and refreshes of ``/etc/issue`` and ``/etc/motd``. The former is displayed in PTY, e.g. whenever a user starts the VM, the latter when logging in via SSH. Both files contain branding, legal notice and currently configured URLs to access the VMMgr web interface. The files are regenerated every time the host is changed via VMMgr or whenever the user presses a key in PTY displaying ``vmtty``.
|
||||
|
||||
The ``/sbin/vmtty``, which is the default *login program* used by `/sbin/getty` defined in `/etc/inittab` and formally not a part of VMMgr, relies on this VMMgr functionality and calls ``/usr/bin/vmmgr rebuild-issue`` to trigger the regeneration. The command is invoked as-is with no parameters or environment variables and loads the settings from VMMgr `configuration`_.
|
||||
|
||||
SSH
|
||||
---
|
||||
|
||||
VMMgr allows to configure *authorized_keys* for SSH access. The file which it displays and modifies is ``/root/.ssh/authorized_keys``. SSH daemon is disabled by default, but once the user enters a SSH keys, VMMgr automatically enables and starts also the daemon, making SSH available on all network interfaces (including `WireGuard`_). Conversely, if the content of the file is emptied via VMMgr's web interface, VMMgr stops and disables SSH daemon.
|
||||
|
||||
WireGuard
|
||||
---------
|
||||
|
||||
Another service which VMMgr allows to configure is WireGuard VPN. WireGuard is both an application and a protocol for point-to-point VPNs, building on modern cryptography algorithms. WireGuard utilizes Curve25519 for key exchange, ChaCha20 for encryption, Poly1305 for data authentication, SipHash for hashtable keys, and BLAKE2s for hashing. It is a default component of linux kernel as of version 5.6. Alpine 3.11 uses kernel 5.4, therefore WireGuard module and tooling needs to be installed separately, which is why VMMgr declares an explicit dependency to it.
|
||||
|
||||
Since WireGuard is easy to configure, but sill versatile, VMMgr leaves most of the configuration up to the user. It creates ``wg0`` and automactically generate the cryptographical key pair. The public key is displayed in the VMMgr GUI and needs to be configured on other machines. The key pair can be also regenerated via GUI. VMMgr restricts WireGuard operations to 172.17.255.0/24 network. The default listening port is 51820/udp. List of peers can be configured using following stanzas
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[Peer]
|
||||
PublicKey = pi1I6pUcjN//s5OEoaGn6bJQyv8RO5w5HjndV97mHWM=
|
||||
AllowedIPs = 172.17.255.12/32
|
||||
Endpoint = 12.34.56.78:51820
|
||||
|
||||
Where the ``PublicKey`` is the public key of the peer / partner, ``AllowedIPs`` is the internal IP range which will be routed to that peer (typically only the peer's VPN IP) and ``Endpoint`` is the IP address and port of peer's WireGuard interface reachable from the internet. For the rest of the settings, refer to the `official WireGuard documentation <https://www.wireguard.com/>`_. The ``[Interface]`` section is hidden in VMMgr.
|
||||
37
doc/toolchain/vmmgr-overview.rst
Normal file
37
doc/toolchain/vmmgr-overview.rst
Normal file
@ -0,0 +1,37 @@
|
||||
VMMgr overview
|
||||
==============
|
||||
|
||||
VMMgr is simply a **Virtual Machine Manager**, a web interface for interaction with the virtual machine and SPOC. It offers user friendly frontend for the commonly used features of the virtual machine.
|
||||
|
||||
VMMgr is a WSGI application written in python 3.7 and running as standalone service employing werkzeug HTTP server. VMMgr is not a mandatory component for SPOC, respective applications and containers, it is only a web interface to conveniently manage the whole virtual appliance.
|
||||
|
||||
Authentication
|
||||
--------------
|
||||
|
||||
VMMgr requires authentication to allow access to most features. There is only a single administrative user whose username and password is the same as for the LUKS disk encryption. The password can be changed any time via the VMMgr interface.
|
||||
|
||||
Portal
|
||||
------
|
||||
|
||||
The main page of VMMgr is called *portal* and contains tiles with information about installed SPOC applications. Unauthenticated user doesn't see the icons and exact names of the applications and sees only the general description. Authenticate used sees all the information, including usernames and passwords generated during the application installation. Application is visible on portal only if it is running and it's visibility is allowed, which is the default.
|
||||
|
||||
Application manager
|
||||
-------------------
|
||||
|
||||
VMMgr allows the user to set up the repository URL, username and password for SPOC. It currently doesn't allow to pin the repository's public key. Once the repository is set, it allows to install, update and uninstall the applications as well as stop or start them and set visibility in portal and select if the application should be automatically start as part of the VM startup.
|
||||
|
||||
There are also the common settings on the same page, which allows to set some common settings important in the context of some applications, such as SMTP sender email or Google API keys.
|
||||
|
||||
Finally, there is a form to change the administrator and disk encryption password and also buttons to power off and restart the VM. Note that the restart requires the disk encryption password to be entered, which can usually be done only directly in the hypervisor console.
|
||||
|
||||
Host settings
|
||||
-------------
|
||||
|
||||
Host settings page contains a wizard to guide the administrator through the process of setting up and verifying the HTTP host settings. The user can set the basic FQDN and the HTTPS port and can verify the DNS settings and reachability from the internet in the next steps.
|
||||
|
||||
There is also a form to work with the HTTPS certificate on this page. The certificate can be either created as self-signed, manually uploaded or automatically requested from Let's Encrypt certification authority using ACME protocol.
|
||||
|
||||
Remote access settings
|
||||
----------------------
|
||||
|
||||
These settings allow to configure remote administration via SSH and remote access via WireGuard VPN. The remote access settings are intended for advanced administration. For more details on usage, see `VMMgr internals <vmmgr-internals>`_ sections `SSH <vmmgr-internals#ssh>`_ and sections `WireGuard <vmmgr-internals#wireguard>`_.
|
||||
Loading…
Reference in New Issue
Block a user