Introduction ¶

We based our development on Python and its scientific ecosystem. On the one hand, this is a great advantage, on the other hand, we needed to work a lot to get good performances. We choose to use a Python-Numpy compiler called Pythran. We even created a new solution called Transonic to help developers to easily use this tool.

Pythran and Transonic are really great for us but using Pythran was also an issue because users needed to compile our packages, i.e. in the end compile C++ files, which lead to all sorts of difficulties and was clearly a barrier for users.

Users need to compile our packages because we did not know how to produce and publish wheels on PyPI. Wheels are archive files containing ready to use packages with compiled extensions. They can be downloaded and installed without compilation with commands like pip install numpy or pip install fluidimage .

Packaging and software engineering ¶

Another important aspect is the build system, i.e. the system which build all extensions by compiling Pythran and C++ files and create a wheel. Historically, we used Setuptools, which was the standard tool. With a lot of pain, we were able to use it to do what was needed to build our packages. The setup was particularly complicated and non standard for Fluidfft, which used C++, Cython and Pythran. This was clearly not a good and maintainable solution.

Hopefully, Python packaging recently improved a lot. The “frontend” (for example pip) now uses a backend (Setuptools, …) to build a Python package in a isolated environment automatically created. Scipy and scikit-image, who also use Pythran, started to use the generalist build system Meson through the meson-python backend.

We now use this build system for packages using Pythran (Fluidsim, Fluidfft and Fluidimage) and for packages using Cython (few fluidfft plugins). However, this was not so simple since we use Pythran through our own package Transonic. Hence, we had to find and implement a good solution to be able to use Transonic with Meson. This development was done with Olivier Giorgis from Logilab. We released 5 versions of Transonic in the beginning of 2024 and I think Transonic is now the easiest way to use Pythran in Python packages and to build them with Meson.

For other Fluiddyn packages, other backends are used, pdm-backend for Fluiddyn and Transonic, and flit_core for fluidfft-builder and fluidfft-mpi4pyfft.

Continuous integration and web based development platform ¶

We started to use continuous integration to run tests since nearly the beginning of the FLuiddyn project. After Bitbucket stopped their support for Mercurial, we migrate our project to Heptapod, a Gitlab fork supporting Mercurial. We are very happy of this move, and Heptapod is much better that Bitbucket in many aspects, in particular in terms of Mercurial support. We fully adopted the workflow advised by Heptapod based on Merge Requests and modern Mercurial features (topics and evolve). I personally think it is much better than the standard Github workflow.

We gradually used more and more the Heptapod continuous integration (CI) for other tasks than pure testing (linting, building the documentation, building wheels, publish on PyPI, etc.). Moreover, we also use Github CI. Anyway, we need to mirror our repositories hosted on Heptapod for some services that do not support Heptapod (for example codecov). So using Github CI is just a matter of having few yaml files in a directory .github/workflows . For packages using Pythran, publishing wheels on PyPI implies building them for the different supported platforms and Python versions. For this task, we choose to follow the main track and do it with cibuildwheel on Github CI. As a result, users on Windows, macOS and Linux can now install Fluidimage with just a pip install fluidimage !

Development, lock files and automation ¶

Another novelty in our development workflow is the generalization of usage of PDM, Nox and Pixi to automate all tasks associated with the development of our packages. For example, we have a Nox session to add a tag to trigger the CI jobs that build and push wheels on PyPI.

With PDM and Pixi, we create and use versioned lock files, which contains the exact versions of the dependencies used for testing. Therefore, we are able to know which dependencies where used for which commits. Also, the dependency resolutions (and hence the potential change of versions of our dependencies) are not done for all runs of our CI but only when we want. This ensures a much better reproducibility and allows us to separate the different tasks, on the one hand, development of new features/fixing bugs and on the other hand simple maintenance to follow and fix the incompatibilities popping up with new versions of our dependencies.

Our development is based on PDM, which creates standard Python environments with packages from PyPI. We also use Pixi for testing with conda-forge environments.

Conclusions ¶

We spent quite a lot of time and energy to improve our development infrastructures and workflows. We reached a very satisfactory state which should help both the increase in usage and the improvements of our packages.