Contributing

Thanks for your interest in contributing to temBoard. temBoard is an open source project open to contribution from idea to code and more.

Submitting an Issue or a Patch

We use the dalibo/temboard GitHub project to track issues and review contributions. Fork the main repository and open a pull request against master branch as usual.

Cloning the Repository

Get temBoard UI and agent sources in one single repository:

$ git clone https://github.com/dalibo/temboard.git
$ cd temboard/

Directories Overview

dalibo/temboard git repository contains a few sub-projects. Here is a quick overview.

  • docs/ - Global mkdocs documentation sources.
  • ui/ - Python Tornado project for temBoard UI aka server.
    • ui/temboardui/toolkit - Shared library between agent and UI.
  • agent/ - Bare Python project for temBoard agent.
    • agent/temboardagent/toolkit - Symlink to toolkit in UI source tree.
  • perfui/ - Docker & Grafana project to visualize temBoard performances traces.
  • dev/ - Development scripts and setup.
  • docker/ - Quickstart Docker Compose file.

Python package is temboardui for temBoard UI and temboardagent for temBoard agent. agent.

Development Requirements

You need the following software to contribute to temBoard:

  • bash, git, make, psql.
  • Docker Compose.
  • Python 3.6 with venv module.
  • NodeJS and npm for building some assets.

Setup Development

Running development version of UI and agent requires two terminals, one for each.

The develop make target creates a virtual environment for Python 3.6, installs temBoard UI, its dependencies, development tools, starts docker services and initializes temBoard database.

$ make develop
make venv-3.6
make[1] : on entre dans le répertoire « /home/.../src/dalibo/temboard »
python3.6 -m venv .venv-py3.6/
...
2020-03-24 17:09:05,937 [30557] [migrator        ]  INFO: Database is up to date.
Initialized role temboard and database temboard.
docker-compose up -d
temboard_repository_1 is up-to-date
Creating temboard_instance_1 ... done
Creating temboard_agent_1    ... done


    You can now execute temBoard UI with .venv-py3.6/bin/temboard


$ .venv-py3.6/bin/temboard --debug
 INFO: Starting temboard 8.0.dev0.
 INFO: Found config file /home/.../temboard/temboard.conf.
 INFO: Running on Debian GNU/Linux 11 (bullseye).
 INFO: Using Python 3.6.8 (/home/.../.cache/pyenv/versions/temboard-uoBqmXGk-py3.6/bin/python) and Tornado 4.4.3 .
 INFO: Using libpq 11.5, Psycopg2 2.8.6 (dt dec pq3 ext lo64) and SQLAlchemy 1.3.24 .
2022-03-16 14:08:06,425 temboardui[1593889]: [pluginsmgmt     ]  INFO: Loaded plugin 'dashboard'.
...
2022-03-16 14:08:06,489 temboardui[1593889]: [temboardui      ]  INFO: Serving temboardui on https://0.0.0.0:8888
...

Go to https://127.0.0.1:8888/ to access temBoard running with your code!

You now need to run the agent. Open a second terminal to interact with the agent and execute the following commands.

$ docker-compose exec agent0 /bin/bash
root@91cd7e12ac3e:/var/lib/temboard-agent# sudo -u postgres hupper -m temboardagent
 INFO: Starting temboard-agent 8.0.dev0.
 INFO: Found config file /etc/temboard-agent/temboard-agent.conf.
2020-08-11 14:29:45,834 [ 3769] [app             ] DEBUG: Looking for plugin activity.
...

The agent is preregistered in UI, using host 0.0.0.0, port 2345 and key key_for_agent. The monitored Postgres instance is named postgres0.dev.

Beware that two Postgres instances are set up with replication. The primary instance may be either postgres0 or postgres1. See below for details.

psql for Monitored PostgreSQL

If you need to execute queries in monitored PostgreSQL instances, execute psql inside the corresponding agent container using the following command:

$ docker-compose exec agent0 sudo -iu postgres psql
psql (13.5 (Debian 13.5-0+deb11u1), server 14.1)
WARNING: psql major version 13, server major version 14.
         Some psql features might not work.
Type "help" for help.

postgres=#

Playing with Replication

Two postgres instances are up with replication. You can execute a second agent for it likewise:

$ docker-compose exec agent1 /bin/bash
root@91cd7e12ac3e:/var/lib/temboard-agent# sudo -u postgres hupper -m temboardagent
 INFO: Starting temboard-agent 8.0.dev0.
 INFO: Found config file /etc/temboard-agent/temboard-agent.conf.
2022-01-11 10:12:55,130 [ 1568] [app             ] DEBUG: Looking for plugin activity.
...

bash history is shared amongst these two containers.

In UI, the seconde agent is pre-registered with address 0.0.0.0, port 2346 instead of 2345, with the same key key_for_agent. The instance FQDN is postgres1.dev.

The script dev/switchover.sh triggers a switchover between the two postgres instances. Executing dev/switchover.sh one more time restore the original typology.

Launching Multiple Agents

Default development environment instanciates two PostgreSQL instances and their temBoard agents. Root Makefile offers two targets to help testing big scale setup :

  • make mass-agents loops from 2347 to 3000 and instanciate a PostgreSQL instance and an agent to monitor it. Each instanciation requires you to type y and Enter. This allows to throttle instanciations and to stop when enough instances are up.
  • make clean-agents trashes every existing instances from 2347 to 3000, without interaction. make clean-agents does not unregister agents!

Choosing PostgreSQL Version

You can change the version of the monitored PostgreSQL instance by overriding image tag in docker-compose.override.yml.

# file docker-compose.override.yml
version: "2.4"

services:
  postgres0:
    image: postgres:9.5-alpine &postgres_image

  postgres1:
    image: *postgres_image

Now apply changes with make develop. Docker-compose will recreate postgres0 and agent0 containers, thus you need to start the agent as documented above.

Note that defining a different major version for postgres0 and postgres1 breaks physical replication.

Execute Unit Tests

Enable the virtualenv and use pytest to run unit tests:

$ . .venv-py3.6/bin/activate
$ pytest ui/tests/unit
...
==== 31 passed, 10 warnings in 1.10 seconds ======
$ pytest agent/tests/unit
...
=============== 6 passed in 0.25s ================
$

Execute Integration Tests

The tests/ directory contains a pytest project to tests both UI and agent using selenium. These tests are not to be confused with agent/tests/func.

Execute these tests right from your virtualenv, using pytest:

$ . .venv-py3.6/bin/activate
$ pytest tests/
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-7.0.1, pluggy-1.0.0 -- /home/bersace/src/dalibo/temboard/.venv-py3.6/bin/python3.6
cachedir: .pytest_cache
postgresql: 14 (/usr/lib/postgresql/14/bin)
sqlalchemy: 1.4.35
system: Debian GNU/Linux 11 (bullseye)
tornado: 6.1
libpq: 14.2
psycopg2: 2.9.3 (dt dec pq3 ext lo64)
temboard: 8.0.dev0 (/home/bersace/src/dalibo/temboard/.venv-py3.6/bin/temboard)
temboard-agent: 8.0.dev0 (/home/bersace/src/dalibo/temboard/.venv-py3.6/bin/temboard-agent)
rootdir: /home/bersace/src/dalibo/temboard/tests, configfile: pytest.ini
plugins: mock-3.6.1, cov-3.0.0, tornado-0.8.1, anyio-3.5.0
...
tests/test_00_setup_ui.py::test_temboard_version PASSED                  [ 12%]
...
tests/test_20_register.py::test_web_register PASSED                      [100%]

============================== 8 passed in 17.69s ==============================
$

pytests tests/ --help describes custom options --pg-version and --selenium. Take care of the custom pytest report header, it shows which temboard and temboard-agent binary is used, the bin directory of PostgreSQL.

pytests tests/ --fixtures describes fixtures defined by tests/conftest.py. Fixtures configure a postgres for monitoring, an agent and the UI in workdir/ prefix.

Selenium standalone container runs a headless Xvfb server with noVNC enabled. View live tests in your browser at http://localhost:7900/ . Click the connect button and interract with the browser and UI.

Selenium container may be flaky. If you suspend your computer, you may have timeout from selenium. Use make restart-selenium to workaround this.

Coding Style

An .editorconfig file configures whitespace and charset handling in various programming language. The EditorConfig site links to plugins for various editors. See .editorconfig for a description of the conventions. Please stick to these conventions.

Python syntax must conform to flake8. CI checks new code with flake8.

UI Database Schema Version

temBoard repository is versionned. A version is the name of a file in temboardui/model/versions. Each file contains the code to execute to upgrade to this version.

To create a new version, put a new file in temboardui/model/versions/ prefixed with a discrete number following the last version. As of now, version file must ends with .sql and contains valid PostgreSQL SQL.

That's all. Use temboard migratedb to check and upgrade temBoard repository.

Building CSS and Javascript

temBoard UI mainly relies on Bootstrap. The CSS files are compiled with SASS. Execute all the following commands in ui/ directory.

In case you want to contribute on the styles, first install the nodeJS dev dependencies:

$ npm install

Then you can either build a dist version of the CSS:

$ grunt sass:dist

Or build a dev version which will get updated each time you make a change in any of the .scss files:

$ grunt watch

Editing Documentation

The documentation is written in Markdown and built with mkdocs.

$ .venv-py3.6/bin/mkdocs serve
INFO     -  Building documentation...
INFO     -  Cleaning site directory
INFO     -  The following pages exist in the docs directory, but are not included in the "nav" configuration:
              - alerting.md
              - postgres_upgrade.md
INFO     -  Documentation built in 0.42 seconds
INFO     -  [16:21:24] Serving on http://127.0.0.1:8000/
...

Go to http://127.0.0.1:8000/ to view the documentation. mkdocs has hot reload: saving file triggers a refresh in your browser.

Building Snapshot Package

You can build a RPM or deb snapshot like this:

$ make dist
$ make -C ui/packaging/rpm build-rhel8
$ make -C agent/packaging/deb build-buster

Find packages in ui/dist or agent/dist directories. See further targets in {ui,agent}/packaging/{deb,rpm}/Makefile.

Releasing

Releasing a new version of temBoard requires write access to master branch on main repository, PyPI project, Docker Hub repository and Dalibo Labs YUM and APT repositories.

To release a new version:

  • Checkout release branch (like v7).
  • Choose the next version according to PEP 440.
  • Update ui/temboardui/version.py and agent/temboardagent/version.py without committing. The version must be the same.
  • Generate and push commit and tag with make release.
  • Push Python eggs to PyPI using make upload.
  • Build and upload Debian and RPM package with make packages.

Throw Development Environment

make clean destroy virtual environments and docker services. Restart from make develop as documented above. If you only need to trash services, use docker-compose as usual : docker-compose down -v, running make develop will restart them and configure the database.