This is the main repository containing all installation scripts, configuration and customization of operating system all application present in Spotter Cluster virtual machine.
## Operating system preparation
### Virtual machine specifications
- **Memory:** 4 GB
- **CPU:** 1 processor, 2 cores
- **Hard Disk:** SCSI, 60 GB
### Operating system media download
Download **Debian 9.0.0 amd64 netinstall** from <https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/> and launch Text-mode installation
## Operating system installation
### Locale settings
- **Language:** English
- **Country, territory or area:** other
- **Continent or region:** Europe
- **Country, territory or area:** Czech Republic
- **Country to base default locale settings on:** United States - en_US.UTF-8
- **Keymap to use:** American English
### Hostname
- **Hostname:** debian
- **Domain name:** [leave blank]
### Users
- **Root password:** [random password of your choice]
- **Full name for the new user:** user
- **Username for your account:** user
- **Choose a password for the new user:** [random passowrd of your choice]
Note: The *user* account will be deleted afterwards as part of installation steps.
### Partitions
- **Partitioning method:** Manual
1. Create empty partition table on SCSI1 (0,0,0) (sda)
<p><strong>Registr kontaktů</strong> asociací, organizací, jednotek zaměstnanců, dobrovolníků, <strong>Registr prostředků</strong>, materiálních zdrojů určených pro činnost v krizových situacích, <strong>logistika</strong> krizového zboží ve skladištích, úkrytech, <strong>organizace lidských zdrojů</strong>, diobrovolníků, <strong>mapová vizualizace</strong> pro lokalizaci a popis krizové události a <strong>mnoho dalších funkcí</strong>.</p>
<p>Mobilní klient k aplikaci Sahana EDEN. Po stažení se zaregistrujte a pro přístup k serveru zadejte IP vaší instalace Sahana EDEN.<br>
<ahref="https://itunes.apple.com/us/app/sambro-mobile/id1127251669"><imgsrc="img/ios.png"class="ico"alt="IOS">IOS 6.0 a vyšší</a><br>
<ahref="https://play.google.com/store/apps/details?id=io.sahana.sambro.mobile"><imgsrc="img/android.png"class="ico"alt="Android">Android 4.0 a vyšší</a>
<ahref="https://play.google.com/store/apps/details?id=org.odk.collect.android"><imgsrc="img/android.png"class="ico"alt="ODK Collect">ODK Collect pro Android</a><br>
<ahref="https://play.google.com/store/apps/details?id=org.redcross.openmapkit"><imgsrc="img/android.png"class="ico"alt="Android">OpenMapKit pro Android 4.1 a vyšší</a>
<p><strong>Dávkový upload videa</strong> do aplikace Pan.do/ra. Podpora rozlišení HD. Videa nahrávejte pouze pod licencí Creative Commons nebo Public Domain.</p>
<ahref="https://play.google.com/store/apps/details?id=eu.it_quality.kanboard"><imgsrc="img/android.png"class="ico"alt="KanBoard">KanBoard client pro Android 4.1 a vyšší</a><br>
<ahref="https://f-droid.org/packages/in.andres.kandroid/"><imgsrc="img/android.png"class="ico"alt="Android">Kandroid pro Android 4.2 a vyšší</a>
<ahref="https://play.google.com/store/apps/details?id=com.mifos.mifosxdroid"><imgsrc="img/android.png"class="ico"alt="KanBoard">Mifos X client pro Android 3.0 a vyšší</a><br>
<p>Info o Misi a Vizi projektu, včetně kontaktu. Zachovejte data bezpečná a neposkytujte je nepovolaným osobám.<br>
<small>CC 4.0 CZ by <ahref="http://trendspotter.cz">TS</a>. Content is based on PD, CC, GNU/GPL. Brand names, trademarks belong to their respective holders.</small>
CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It is written in python, has Flask-based web interface and uses Postgres database with PostGIS extension. CKAN exposes API which allows third parties to create extensions.
The project is split to several components. The most important ones are:
- **ckan** - https://github.com/ckan/ckan - The main DMS component with Flask frontend.
- **datapusher** - https://github.com/ckan/datapusher - A standalone web service that pushes data files from a CKAN site's resources into its DataStore.
- **ckan-service-provider** - https://github.com/ckan/ckan-service-provider - A library for making web services that make functions available as synchronous or asynchronous jobs.
Apart from that, CKAN requires a *Redis* in-memory data store and *Solr* search platform instance. Note that CKAN is shipped with Solr schemas which don't work well with up-to-date versions of Solr, therefore we use older Solr 6.5.1.
DataPusher
----------
DataPusher is running as a separate service. Its development is a bit out of sync with the core ckan component. SKAN has already moved to python 3, however datapusher is still working only with python 2.7. Due to that, DataPusher has its own image and is running as a separate container.
Alpine, being a progressive linux distribution, stopped supporting uwsgi connector for python 2.7 in Alpine 3.11, therefore the container needs to be based on older Alpine 3.10 which still ships with *uwsgi-python2*.
The connection between CKAN and DataPusher is done via HTTPS endpoints. There is a possibility to disable TLS certificate verification, but a better solution seems to be import any self-signed certificate into the DataPusher's trust store. There is a script which automatically handles the import whenever the DataPusher container starts.
Extensions
----------
CKAN container comes with several preinstalled and preconfigured extensions. Historically, there were more extensions, but over the time CKAN's API evolved and the development has moved to python 3, so majority of the historical extensions are no longer operational.
Adds Line, Bar and Pie charts. It uses Flot Charts, which is compatible with all major browsers (including IE6+).
ckan/ckanext-spatial
^^^^^^^^^^^^^^^^^^^^
- URL: https://github.com/ckan/ckanext-spatial
Contains plugins that add geospatial capabilities to CKAN. E.g. a spatial field on the default CKAN dataset schema which allows to perform spatial queries and to display the dataset extent on the frontend.
ckan/ckanext-geoview
^^^^^^^^^^^^^^^^^^^^
- URL: https://github.com/ckan/ckanext-geoview
Contains view plugins to display geospatial files and services in CKAN. It contains an OpenLayers based viewer and other view plugins that used to be part of ckanext-spatial.
ckan/ckanext-mapviews
^^^^^^^^^^^^^^^^^^^^^
- URL: https://github.com/ckan/ckanext-mapviews
Adds regular and choropleth maps to CKAN, using the new CKAN Resource View. It uses LeafletJS, which is compatible with all major browsers (including IE7+).
Allows group admin to export datasets from their organizations to export all datasets to a .xls worksheet. Sysadmins may choose to export datasets from any organization they want or to export all datasets of all groups.
Crisis Cleanup is a collaboration platform that connects disaster recovery organization volunteers with people who need help after a disaster. It is a Ruby on Rails application written for Ruby 2.6 with NodeJS-dependent web frontend.
As of 2020-03-27, CrisisCleanup v2 is no longer maintained. The maintenance has been discontinued in favor of CrisisCleanup v3, which splits the backend and frontend into two separate projects. Unfortunately the source of the backend part hasn't been opened and the creator has no plants to do so, so currently only the frontend application as available on https://github.com/CrisisCleanup/crisiscleanup-3-web.
Version 3 uses OpenStreetMap and Leaflet, whereas version 2 used Google Maps.
NodeJS compatibility
--------------------
The application uses NodeJS to generate and pack assets, however due to its age, yarn dependencies installation fails on NodeJS versions newer than 8, therefore the NodeJS version has been frozen on 8.14.0 provided by Alpine 3.8.
Decidim is a participatory democracy platform for cities and organizations. Decidim is written and runs on Ruby 2.6, where it is installed as Ruby Gem. Incoming HTTP requests are handled via passenger nginx plugin.
Upstream URL: https://github.com/decidim/decidim
Passenger nginx plugin
----------------------
The passenger plugin work as application proxy between HTTP server and Ruby on Rails applications. Nginix doesn't have modules API like Apache does, so all nginx module must be present at compile time and only then they can be loaded and unloaded at runtime. Passenger plugin is not present in the standard Alpine nginx packages, therefore for passenger to work, a custom version of nginx including the passenger plugin needs to be compiled. Passenger sources contain ``passenger-install-nginx-module`` script which eases and semi-automates the compilation.
Database upgrades
-----------------
Decidim creates a series of database migration scripts under ``db/migrate`` directory. Unfortunately, these script are not created deterministically as their file names are always created with current build's timestamp. The same script using the same Decidim version therefore always ends up with different database migration scripts. Newly added scripts in a new version of Decidim are not sorted at the end, so it's impossible to determine which scripts need to applied from the filenames alone.
During manual upgrade, the workaround is to compare the directories between the old and the new image, stripping the timestamp and running only the scripts which don't exist in the old directory, however this is impossible to be done during automatic upgrade, as the old image is removed before the upgrade script is executed.
Additional site creation
------------------------
Decidim allows to create managed sites using the superadmin interface on ``https://<Decidim URL>/system/``. The sites require their own (sub)domains, therefore some additional considerations must be taken when setting up HTTP reverse proxies and TLS certificates.
FrontlineSMS is a system for both manual and automatic processing of SMS, supporting creation of various SMS workflows. The application is written in Java (resp. Groovy) and uses Grails layout engine. It has its own installer which deploys single-tenant Jetty lightweight applicaiton server. To work efficiently, FrontlineSMS requires a device with SMS capabilities or a connection to third-party service such as Twilio.
The version of FrontlineSMS is no longer maintained by the original developers and no new updates are expected to appear. The manufacturer currently offers and supports only a cloud version of FrontlineSMS.
Fork URL: https://github.com/trendspotter/frontlinesms2 - the fork contains the last known working version of installer as a release.
Modem capabilities
------------------
There are special considerations to be taken whenever an application witch direct access to hardware is run in a container. Currently FrontlineSMS has not been tested with such device and it is expected that some extra work or container configuration not present in any other application needs to be done in order to make the hardware device available and usable by the container.
Localization
------------
FrontlineSMS is localized to Czech only locally on SpoterVM. Upstream doesn't contain the localization as the application is not developed anymore. The localization is supplied as a Java property file ``WEB-INF/grails-app/i18n/messages_cs.properties`` and then once again as the same file packed and minifies as ``assets/i18n/frontlinesms-core_messages_cs.js``. The minified file can be created using ``frontlinesms-lang-js.py`` helper script in SpotterVM repository.
Java support
------------
Due to the age of the application (latest release somewhere in 2014), FrontlineSMS doesn't work well with current versions of Java runtime. The last know working version is Java 8.222.10 supplied by Alpine 3.8. More recent version, e.g. Java 8.242 cause HTTP connection accept failures.
X server
--------
Although FrontlineSMS is a web application, it requires X server as it attempts to install a systray icon. On a headless server, this requirement can be worked around using ``xf86-video-dummy`` package which works as a *blackhole* display driver.
Kanboard is a Kanban project management tool written in PHP. It supports creation of projects and their population with tasks, provides Kanban boards with swimlanes, charts etc. It is extensile via plugins.
Kanboard is localized via PHP files containing associative arrays with original English string and the translation in the respective language. Every plugin has its own localization files. Due to this, we currently maintain forks or 4 repositories with plugins where the original maintainer decided to discontinue support.
Plugins
-------
Following list summarizes state of Kanboard plugins used or requested in our container as of 05/2020.
Supplies translation strings via additional configuration file for other plugins which are missing or have incorrect translations. Not updated anymore, however the maintainer is active on GitHub. This plugin has initially been requested to be included in our installation, but due to its purpose, it's mostly useless as all translations were done in the respective plugin localization files.
This plugin adds a cover image function to tasks on the board, and allows you to set a project image to use as a logo for projects. Development is active.
Allows to create budget lines, see expenses based on sub-task time tracking and manage user hourly rates. Repository has been forked. The original repository is archived.
Allows to use text messages (SMS) instead of the default TOTP system (Time-based One-time Password Algorithm). Works with Twilio service. Repository has been forked. The original repository is archived.
Adds new date field for tasks to define the visibility on the board and dashboard. Repository is archived. There are no strings for translation, so it hasn't been forked.
1. Used by CKAN extensions *reclineview*, *spatial* and *geoview*.
2. Untested as the EcoGIS source code is not fully open. Looks like the data sources are configurable, but the full documentation is only in Italian.
3. Used by Odoo *Google Maps* module to display company/partner address on map.
4. Map is used by OMK and ODK Android clients. OMK Server only offers API.
5. Sahana Eden supports multitude of connectors and protocols to process map and feature data. ArcGIS REST, Bing maps, GeoJSON, GeoRSS, Google Maps, GPX, KML, MGRS, OSM, OWM, Shapefile, TMS, WFS, WMS and XYZ.
OpenDataKit is a suite of tools that help organizations collect and manage data, typically via mobile application. The most interesting components used on SpotterVM are
- **ODK Aggregate** - Main data storage server and analysis tool, written in Java, running on Apache Tomcat using PostgreSQL database.
- **ODK Build** - Drag-and-drop form designer, written in Ruby 2.4, requires NodeJS for frontend assets.
- **build2xlsform** - Service to convert ODK Build forms to xlsform-compliant XLSX, written in LiveScript (NodeJS).
ODK Build runs in a container separate from ODK Aggregate and doesn't use any database engine for its persistent storage. ODK Build can however export the designed form to ODK Aggregate via HTTP endpoint. In case HTTPS is used with self-signed certificate, it needs to be imported to OpenSSL trust store prior to first communication attempt. In our container, this is ensured by ``add-ca-cert`` script.
build2xlsform lives in the same container as ODK Build and allows for export of ODK Build JSON forms to XLSX files editable via any modern office suite.
OpenMapKit is a standalone extension to OpenDataKit providing mapping features and handles collection and aggregation of OpenStreetMap and OpenDataKit data generated via mobile applications.
NodeJS compatibility
--------------------
OpenMapKit is purely NodeJS application. Unfortunately it depends on a libxmljs module in version incompatible with NodeJS version newer that 10. Due to that OpenMapKit is currently frozen on NodeJS 10.19.0 provided by Alpine 3.10.
Pan.do/ra is a free, open source media archive platform. It allows to manage large, decentralized collections of video and collaboratively create metadata and time-based annotations. Pan.do/ra is a Django-based application written in python and is split to several components, most notably OxJS javascript framework and oxtimelines. It heavily relies on *ffmpeg* and uses *RabbitMQ* to pass messages between the components.
Czech localtization for Pan.do/ra currently exists only in SpotterVM repository. Separate localization is required for Pan.do/ra and OxJS framework which it uses. Moreover, the list of supported languages is hardcoded, so localization entails also modification of this fixed list of languages.
Python 3.8 compatibility
------------------------
Pandora uses Celery message broker, however depends on a version with known incompatibility with python 3.8, see https://github.com/celery/celery/issues/5761. This can be worked around by setting environment variable ``COLUMNS`` to value ``80`` prior to launching the celery process. The affected services run by container's s6 daemon supervisor are currently configured to use this workaround.
Sahana Eden is a flexible and extensible humanitarian platform written in python, running on web2py platform and PostgreSQL with PostGIS extension. Sahana Eden is extremely highly customizable and configurable which unfortunately often leads to various bugs and misunderstandings in the configuration. For the description of the individual configuration options and their effect, see `Sahana Eden configuration usability report <sahana-configuration-report.html>`_. The application is usually configured via templates, which contain all the configuration, forms and logic customizations and some basic data relevant for the intended deployment.
The application supports various interfaces, protocols and formats for data exchange - CAP (Common Alerting Protocol), XML/Atom feeds, WMS, WFS, WCS and other map services, KML exports, XML and CSV tabular data etc. Maps can be configured and adjusted to use virtually any tile, layer and feature sets. See `map services <map-services.html>`_.
Upstream URL: https://github.com/sahana/eden
Setup module
------------
Sahana deployment functions are based on Ansible. Ansible is provisioning, configuration management, and application-deployment tool written in python. It uses YAML files to define the desired actions or states. These files are called *playbooks*. There is a whole separate sahana_deploy project focusing on rapid Sahana Eden deployment and parts of it are used also in the Setup module. The setup module allows to control Sahana instances and adjust their configuration in terms of setting simple values and enabling/disabling modules in ``000_config.py``. The module, as well as the Ansible playbooks for the whole deployment are a bit too strict when it comes to assumptions and discovery of Sahana's environment. E.g. the path to the instance is expected to be always under ``/home/prod``. The Setup module is enabled and can be used, however due to its limitations and expectations it sets, it can't be considered reliable in the environment of Alpine container.
Python 3.8 compatibility
------------------------
Sahana Eden is currently recommended to be run on web2py R2.18.5 (commit hash `59700b8 <https://github.com/web2py/web2py/commit/59700b8>`_). Recommended version is tracked in eden_deploy project in `main.yml for common playbook <https://github.com/sahana/eden_deploy/blob/master/roles/common/tasks/main.yml>`_. This version however doesn't fully support python 3.8, so a series of patches needs to applied to achieve the full support for python 3.8. This includes compatibility patches for gluon and pydal tracked in `_compat.patch <https://github.com/sahana/eden_deploy/blob/master/roles/common/files/_compat.patch>`_ and scheduler patch tracked separately in `scheduler.diff <https://github.com/sahana/eden_deploy/blob/master/roles/common/files/scheduler.diff>`_.
Localization
------------
Czech localization for Sahana Eden is currently not a part of upstream code. It is currently held in SpotterVM repository and there are currently no plans to submit a pull request with the translation.
The localization is done via translation export function of Sahana Eden found in the administration menu. The exported CSV or XLS file is then manually translated and fed back to ``sahana-lang-convert.py`` helper script which generated the appropriate python dictionary file used as translation. Some strings are not output by the translation function and are either hardcoded or composed, so there is also ``sahana-lang-compare.py`` helper script which traverses installed Sahana Eden instance and tries to locate and supply all missing strings from the previous translation file. Both helper scripts are also contained in the SpotterVM repository.
Local development
-----------------
Since Sahana Eden is the crown jewel of the whole environment and unfortunately also the most buggy application, it may be needed to work on some development tasks and bugfixes outsite of the VM environment. Following are a few tips to important tasks and gotchas you may encounter.
To enable debug mode for Sahana, set ``settings.base.debug = True`` in ``applications/eden/models/000_config.py``.
Database migration
^^^^^^^^^^^^^^^^^^
New commits (both upstream and local) can require a modification of database structure. If this is not done, Sahana may report internal server errors or corrupt the database. Migration to the up-to-date schema version can be achieved via following steps.
..code-block:: bash
# Change directory to web2py root
cd /srv/web2py
# Allow database migration (normally denied in prod)
sed -i 's/settings.base.migrate = False/settings.base.migrate = True/g' applications/eden/models/000_config.py
# Remove old cached bytecode
rm -rf applications/eden/compiled
# Run the migration task
python web2py.py -S eden -M -R applications/eden/static/scripts/tools/noop.py
# Deny database migration (improves performance)
sed -i 's/settings.base.migrate = True/settings.base.migrate = False/g' applications/eden/models/000_config.py
# Precompile fresh bytecode
python web2py.py -S eden -M -R applications/eden/static/scripts/tools/compile.py
Factory reset
^^^^^^^^^^^^^
In case you'd like to wipe all data and restore your Sahana instance to pre-installation phase, stop all Sahana (uWSGI) processes and run the following.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.