Thanks for your interest in contributing to temBoard. temBoard is an open source project welcoming contribution from idea to code and more.
Submitting an Issue or a Patch¶
We use the dalibo/temboard GitHub project to track issues and review
contributions. Fork the main repository and open a pull request against
master branch as usual.
Cloning the Repository¶
Get temBoard UI and agent sources in one single repository:
dalibo/temboard git repository contains a few sub-projects. Here is a quick overview.
docs/- Global mkdocs documentation sources.
ui/- Python Tornado/Flask project for temBoard UI aka server.
ui/temboardui/toolkit- Shared library between agent and UI.
agent/- Bare Python project for temBoard agent.
agent/temboardagent/toolkit- Symlink to toolkit in UI source tree.
dev/- Development scripts and data.
docker/- Quickstart Docker Compose file.
tests/- Functional integration tests.
Python package is
temboardui for temBoard UI and
temboardagent for temBoard
You need the following software to develop temBoard:
- bash, git, make, psql.
- Docker Compose.
- Python 3.6 with
- NodeJS 16+ and npm for building some browser assets.
Running development version of UI and agent requires two terminals, one for each.
develop make target
creates a virtual environment for Python 3.6,
installs temBoard UI, its requirements, development tools,
builds agent Docker image,
starts docker services and initializes temBoard database.
$ make develop make venv-3.6 make : on entre dans le répertoire « /home/.../src/dalibo/temboard » python3.6 -m venv dev/venv-py3.6/ ... 2020-03-24 17:09:05,937  [migrator ] INFO: Database is up to date. Initialized role temboard and database temboard. docker-compose up -d temboard_repository_1 is up-to-date Creating temboard_instance_1 ... done Creating temboard_agent_1 ... done You can now execute temBoard UI with dev/venv-py3.6/bin/temboard $ dev/venv-py3.6/bin/temboard --debug INFO: Starting temboard 8.0.dev0. INFO: Found config file /home/.../temboard/temboard.conf. INFO: Running on Debian GNU/Linux 11 (bullseye). INFO: Using Python 3.6.8 (/home/.../.cache/pyenv/versions/temboard-uoBqmXGk-py3.6/bin/python) and Tornado 4.4.3 . INFO: Using libpq 11.5, Psycopg2 2.8.6 (dt dec pq3 ext lo64) and SQLAlchemy 1.3.24 . 2022-03-16 14:08:06,425 temboardui: [pluginsmgmt ] INFO: Loaded plugin 'dashboard'. ... 2022-03-16 14:08:06,489 temboardui: [temboardui ] INFO: Serving temboardui on http://localhost:8888 ...
Go to http://localhost:8888/ to access temBoard running with your code!
You now need to run the agent. Open a second terminal to interact with the agent and execute the following commands.
$ docker-compose exec agent0 /bin/bash root@91cd7e12ac3e:/var/lib/temboard-agent# sudo -Eu postgres temboard-agent INFO: Starting temboard-agent 8.0.dev0. INFO: Found config file /etc/temboard-agent/temboard-agent.conf. 2020-08-11 14:29:45,834 [ 3769] [app ] DEBUG: Looking for plugin activity. ...
The agent is preregistered in UI, using host
0.0.0.0 and port
monitored Postgres instance is named
Beware that two Postgres instances are set up with replication. The primary instance may be either postgres0 or postgres1. See below for details.
.editorconfig file configures whitespace and charset handling in various
programming language. The EditorConfig
site links to plugins for various editors. See
.editorconfig for a
description of the conventions. Please stick to these conventions.
Python syntax must conform to flake8. CI checks new code with flake8.
Executing in debug mode¶
temboard and temboard-agent commands has a debug mode. In debug mode, logs are verbose, files change triggers an automatic restart of the process, an unhandled exception drops in an interactive PDB debugger prompt.
Enable debug mode by setting DEBUG=y environment variable. For agent, only long running commands have autoreload.
psql for Monitored PostgreSQL¶
If you need to execute queries in monitored PostgreSQL instances, execute psql inside the corresponding agent container using the following command:
$ docker-compose exec agent0 psql psql (13.5 (Debian 13.5-0+deb11u1), server 14.1) WARNING: psql major version 13, server major version 14. Some psql features might not work. Type "help" for help. postgres=#
Playing with Replication¶
Two postgres instances are up with replication. You can execute a second agent for it likewise:
$ docker-compose exec agent1 /bin/bash root@91cd7e12ac3e:/var/lib/temboard-agent# sudo -Eu postgres temboard-agent INFO: Starting temboard-agent 8.0.dev0. INFO: Found config file /etc/temboard-agent/temboard-agent.conf. 2022-01-11 10:12:55,130 [ 1568] [app ] DEBUG: Looking for plugin activity. ...
bash history is shared amongst these two containers.
In UI, the second agent is pre-registered with address 0.0.0.0 and port 2346 instead of 2345.
Second instance FQDN is
dev/switchover.sh triggers a switchover between the two postgres
dev/switchover.sh one more time restore the original
Testing previous version¶
Compose project for development configures a stable agent named
This agent is preregistered in development UI.
postgres-stable instance in UI to ensure temBoard UI is compatible with stable agent.
Access Postgres instance monitored by stable agent using the following compose invocation:
$ docker-compose exec agent-stable psql psql (13.5 (Debian 13.5-0+deb11u1), server 13.7) Type "help" for help. postgres=#
Choosing PostgreSQL Version¶
You can change the version of the monitored PostgreSQL instance by overriding
image tag in
Now apply changes with
make develop. Docker-compose will recreate
agent0 containers, thus you need to start the agent as documented above.
Note that defining a different major version for postgres0 and postgres1 breaks physical replication.
Launching Multiple Agents¶
Default development environment instanciates two PostgreSQL instances and their temBoard agents. Root Makefile offers two targets to help testing big scale setup :
make mass-agentsloops from 2348 to 3000 and instanciates a PostgreSQL instance for each number and an agent to monitor it. Number is used as agent port. Each instanciation requires you to type
yand Enter. This allows to throttle instanciations and to stop when enough instances are up.
make clean-agentstrashes every existing instances from 2348 to 3000, without interaction. make clean-agents does not unregister agents!
Execute Unit Tests¶
Each UI and agent projects has its own unit tests battery. Enable the virtualenv and use pytest to run unit tests:
$ . dev/venv-py3.6/bin/activate $ pytest ui/tests/unit ... ==== 31 passed, 10 warnings in 1.10 seconds ====== $ pytest agent/tests/unit ... =============== 6 passed in 0.25s ================ $
Execute Integration Tests¶
tests/ directory contains a pytest project to tests UI and agent
integration using Selenium.
On Debian your UNIX user must be in the ssl-cert group to run the tests. Be careful, the tests will use the local installation of temboard if it exists instead of dev files. To run the tests locally it is better not to have temboard installed.
Execute these tests right from your virtualenv, using pytest:
$ . dev/venv-py3.6/bin/activate $ pytest tests/ ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-7.0.1, pluggy-1.0.0 -- /home/bersace/src/dalibo/temboard/dev/venv-py3.6/bin/python3.6 cachedir: .pytest_cache postgresql: 14 (/usr/lib/postgresql/14/bin) sqlalchemy: 1.4.35 system: Debian GNU/Linux 11 (bullseye) tornado: 6.1 libpq: 14.2 psycopg2: 2.9.3 (dt dec pq3 ext lo64) temboard: 8.0.dev0 (/home/bersace/src/dalibo/temboard/dev/venv-py3.6/bin/temboard) temboard-agent: 8.0.dev0 (/home/bersace/src/dalibo/temboard/dev/venv-py3.6/bin/temboard-agent) rootdir: /home/bersace/src/dalibo/temboard/tests, configfile: pytest.ini plugins: mock-3.6.1, cov-3.0.0, tornado-0.8.1, anyio-3.5.0 ... tests/test_00_setup_ui.py::test_temboard_version PASSED [ 12%] ... tests/test_20_register.py::test_web_register PASSED [100%] ============================== 8 passed in 17.69s ============================== $
pytests tests/ --help describes custom options
--selenium. Take care of the custom pytest report header, it shows which
temboard and temboard-agent binary is used, the bin directory of PostgreSQL and
pytests tests/ --fixtures describes fixtures defined by tests/conftest.py.
Fixtures configure a postgres for monitoring, an agent and the UI in
prefix. This may help you write a new test.
Selenium standalone container runs a headless Xvfb server with noVNC enabled. View live tests in your browser at http://localhost:7900/ . Click the connect button and interract with the tested UI using the embedded Firefox.
Selenium container may be flaky. If you suspend your computer, you may have
timeout from selenium. Use
make restart-selenium to workaround this.
UI Database Schema Version¶
temBoard database is versionned.
A version is the name of a file in
Each file contains the code to execute to upgrade to this version.
To create a new version, put a new file in
prefixed with a discrete number following the last version. As of now, version
file must ends with
.sql and contains valid PostgreSQL SQL.
That’s all. Use
temboard migratedb to check and upgrade temBoard repository.
temBoard UI mainly relies on Bootstrap. The CSS files are compiled with SASS.
ViteJS manages assets from
ui/temboardui/static/src. ViteJS builds assets in
Execute all the following commands in ui/ directory.
In case you want to contribute on the styles, first install the NodeJS dev dependencies:
Then you can either build a dist version of the CSS:
Or run dev server which watch changes on source files and hot-reload modules in your browser:
Now restart temBoard UI configured with ViteJS dev server base URL from
$ VITEJS=http://localhost:5173 temboard serve ... 2022-08-23 10:40:57 CEST temboardui DEBUG: vitejs: Using ViteJS dev server at http://localhost:5173. 2022-08-23 10:40:57 CEST temboardui DEBUG: vitejs: Skip reading ViteJS manifest. ...
Beware of CORS! Depending on your browser product and version, you may hit
an unstyled page if CORS policy denies loading assets from ViteJS dev server.
Ensure you run temBoard in plain HTTP and that you access both temBoard and
ViteJS dev server on
All assets managed by ViteJS are hot reloaded, including CSS. ViteJS Hot reload does not required reloading server-side.
The documentation is written in Markdown and built with
Editing documentation requires Python 3.7.
$ dev/venv-py3.7/bin/mkdocs serve INFO - Building documentation... INFO - Cleaning site directory INFO - The following pages exist in the docs directory, but are not included in the "nav" configuration: - alerting.md - postgres_upgrade.md INFO - Documentation built in 0.42 seconds INFO - [16:21:24] Serving on http://127.0.0.1:8000/ ...
Go to http://127.0.0.1:8000/ to view the documentation. mkdocs has hot reload: saving file triggers a refresh in your browser.
Try to use semantic line breaks: split lines by idea instead of reflowing words. This helps reading diff, handling conflicts when rebasing.
Building RHEL Package¶
Building RPM packages for RHEL and compatible clones requires Docker and Docker Compose for isolation. Uploading to Dalibo Labs requires internal project yum-labs and access.
UI and agent each has
packaging/rpm directory with a Makefile and scripts to build RPM packages.
build-rhel<version> make target like this:
Version can be either 9, 8 or 7.
agent/packaging/rpm/Makefile provides the same targets.
The builder script searches for wheels in
and if not found, tries to download wheel from PyPI.
Use top level
make dist to generate wheels.
Building Debian Package¶
Building debian packages requires Docker and Docker Compose for isolation. For
signing, you need the
devscripts package and a GPG private key. For
uploading, you require
Define environment variables
DEBEMAIL. mkchanges.sh scripts
signs changes with your GPG key matching these environment variables.
Each UI and agent has
packaging/deb/ directory with a Makefile and scripts to
build packages. Use
build-<codename> target like this:
codename is one of
agent/packaging/deb/Makefile provides the same targets.
The builder script search for wheels in
and if not found, tries to download wheel from PyPI.
Use top level
make dist to generate wheels.
Testing with Grafana¶
temBoard exposes /metrics in OpenMetric format. Development environment includes Prometheus and Grafana preconfigured with authenticated access and file-based service discovery. You only need a couple of steps to view temBoard metrics in Grafana.
- Start development temBoard UI and at lease one of agent0 or agent1.
- Generate Prometheus targets using
- Go to development Grafana import page.
- Import Dashboard ID 1860 (Node Exporter Full) or 9628 (PostgreSQL Database).
- Now wait a few minutes that Prometheus scrapes data. Scrape interval is 1 minute.
temBoard exposes only a subset of metrics from node_exporter and
postgres_exporter, plus a set of custom metrics prefixed with
xpg_wal_files_bytes_total. Thus a lot of panel will show N/A.
Prometheus also watch
dev/prometheus/targets/custom.yaml files for hand
Releasing a new version of temBoard requires write access to master branch on main repository and Dalibo Labs YUM and APT repositories.
To release a new version:
- Checkout release branch : master for v8.
agent/temboardagent/version.pywithout committing. The version must be the same and follow PEP440.
- Check, commit, tag and push using
For stable release, you need write access to Dalibo Labs repositories:
- Wait for CircleCI pipeline to publish GitHub releases.
- Download packages with
- Publish Debian and RPM packages with
To release a v7 minor version, please follow v7 documentation.
make clean destroy virtual environments and docker services. Restart from
make develop as documented above. If you only need to trash services, use
docker-compose as usual :
docker-compose down -v, running
make develop will
restart them and configure the database.