bookmark_borderA wicked problem from the past

In the last few evenings there was great despair. Trying to solve one problem would reveal yet another one. As a result, I hesitated to release some of the changes immediately. I don’t want to make people suffer, having to deal with new improved problems of the day whenever they run an update on their horde installations. I’m glad feedback on the mailing list has improved quite a lot, sometimes coming with patches and suggestions and even people telling me things start to just work. That’s great. But whenever you see the light, there’s a new challenge on the horizon. Or rather something very old lurking in the shadows.

Break it till you make it

So recently we felt confident enough to switch our local installation from a frozen state to the latest version of the wicked wiki and the whups bug tracker. We also updated the PHP container, the database version and the likes. This turned into a great opportunity to test the rollback plan. 🙄 Issues were cascading.

Bullets you might have dodged

Generally having both the composer autoloader and the horde autoloader work together creates some friction. This is complicated when the exact order in which they are initialized is not the same in all scenarios. As reported previously, Horde’s apps are not strictly conforming the PSR-0 autoloading standard while the libraries mostly do. Newer releases of the apps autoload using a class map instead of PSR-0 logic. But that class map was not restricted enough in the first iterations. Thus you might have the canonical locations of classes in the autoloader, but also the version in the /web/ dir or, in case of custom hook classes, the version from the /var/config/ directory. Torben Dannhauer suggested to restrict the rules and we will do that with the next iteration. The first attempt to fix this unfortunately broke overriding the mime.local.php config file. An update is in the git tree and will be released as a version later this week. But I’m glad we ran into this as it revealed another dark secret

We part, each to wander the abyss alone

Turned out the wicked wiki software carried a little library Text_Wiki in its belly. Hailing from PHP 4 days, it’s archaic in its structure and its way of implementing features. Parts of the test suite predate phpunit and the way it’s packaged is clearly not from our times. This library also exists as a separate package horde/text_wiki. Which also exists as pear/text_wiki both in the classic PEAR archive and the modern github repository. While the base package does not exist in packagist, the extension driver for mediawiki format does. Odd enough. It’s really a shame the software from the old PEAR ecosystem is slowly fading away because of its ossification. They all share ancestry and they were all largely written and maintained by the same set of people but they are all different in subtle ways.

Text_Wiki works great when it works. But it’s a real treasure trove of incompatibilities with modern PHP versions. Over the years, unicode support has evolved along with the strictness of PHP’s core. While I am very much willing to contribute back any changes to the official pear version, I have my doubts if they will be accepted or handled at all.

Rising again like the phoenix

I really, really did not want to deal with any of this. Creating a fourth version out of three splinters feels like madness. But what can you do?
The result is an upcoming version of horde/text_wiki and horde/wicked which is a radical break. The new text/wiki has no PSR-0 lib/ content. It’s all PSR-4, it’s namespaced, every single class name and file name has changed. A new test suite has been started but only a tiny fraction of unit tests have been ported so far. The different extension drivers for mediawiki and tiki dialects have been incorporated into the base package as I did not want to maintain a hand full of packages.

The whole thing couples as little to horde as possible. It has its own exceptions based on the PHP builtin exception tree, so no dependency on horde/exception. I even stripped the dependency on Pear_Exception as there is no value in keeping that nowadays. Anything which ties this revamped library into the horde ecosystem now lives in horde/wicked. Extra markup rules which render horde portal blocks into a wiki page. Adapters querying the database if a linked page already exists or must be created. Dynamic links created from the registry and application states. None of this glue belongs into the text_wiki library.

Many incompatibilities with modern PHP have been solved. Often this was a matter of mixing and matching bits from the three different splinter versions out in the wild. Some of it is just a chore to do. Of course, the pear XML files are gone and the composer file is fully compliant with packagist and most recent composer. At least this part has been contributed back already.

Dawn, finally

It will be another few evenings until the new horde/wicked and the new horde/text_wiki are ready for a tagged release, probably along with some of the changes to other libraries I explained above. There’s probably something that will break and need fixing. But that shouldn’t block progress for too long.

bookmark_borderProxy Mode in horde-installer-plugin

Horde 5 relied on the PEAR ecosystem ripping apart their packages and putting everything where it needs to be. But PEAR is a thing of the past. Horde 6 integrates with the composer installer using an additional piece called the horde/horde-installer-plugin. This composer plugin links together pieces from the installed composer packages into the right places and adds auto-generated configuration to make sure everything is found even though stuff is NOT mixed together the way PEAR did. This was a good starting point early in Horde 6 development when we took great pains not to break the previous distribution installs (which relied on PEAR or PEAR like behaviour) and installs based on horde-git-tools.

However, by now this is much less a priority and we are absolutely willing to move forward and make things less complicated, potentially breaking and changing code that relied on these older schemes. Much less code and assets are going to be exposed through the /web/ folder.

Proxy Mode vs Symlink Mode

Version 2.7.0 of horde/horde-installer-plugin introduces a new switch to the horde:reconfigure command:

composer horde:reconfigure --mode=proxy
  • In proxy mode composer will not symlink whole apps to the web dir but only create folders and files outside lib/, src/, test/ and a fairly large exclusion list. The endpoint php files will forward requests to the actual code in /vendor dir. This should reduce code duplication and makes it more obvious where related files need to be looked up
  • A new constant HORDE_CONFIG_BASE is written to the autogenerated registry snippets.
  • The composer autoloader is directly loaded by the proxy php files before delegating to the actual file in the vendor dir.
    If you see unexpected class not found messages when testing proxy mode, please upgrade your horde/core package.

Currently proxy mode is a one-off. However, in an upcoming version the last used mode will be persisted to the composer.json file and reused on subsequent commands without an explicit mode option. Proxy mode is supposed to become the default in horde-installer-plugin 3.0.

Benefits of proxy mode

Proxy mode is the first step towards clarifying the intended architecture. Until recently much of our detection logic needed to behave differently if called through a file in the webroot or the same file but in its original vendor/horde/$app location. The new proxy files always perform the same way:

  • First load the composer autoloader because the file always knows where the autoloader is relative to its own path.
  • Then load the actual file in the vendor/horde/$app location
  • Horde’s own internal PSR-0+ autoloader would only be loaded after composer’s autoloader. It hits action less and less often and will probably be removed from the default setup altogether.

This means the actual class files can rely on __FILE__ pointing to a predictable scheme and much less guess work and protection code is needed. On the other hand, it reveals some hidden problems and complications which developers can now fix before making this new mode the default. Unexpected lowercase class names (as found in whups and Horde Forms), complicated chicken/egg issues in the Registry, duplicate class names and parts of the error handling broken by subtle changes in newer PHP versions can now be seen and fixed.

As a consequence of spotting bugs and loading issues as I go along, starting with version 2.7.1 the composer plugin will check if a file var/config/autoload-extra.php exists. If it does, it will be included after the composer autoloader. This allows selectively hardcoding some class files until fixed versions of apps become available – or to actively undefine something for tests.

bookmark_borderUnit Testing Horde 6

Some years ago I reasoned about upgrading unit tests from ancient PHPUnit 4 to then-recent PHPUnit 9.
Back then Horde’s unit test suite would use a Horde_Test class inheriting from the actual phpunit test cases. Even then I was fairly certain that this approach was not very practical in the long run.
Why extending PHPUnit might be wrong – ralf-lang.de

Turns out this is right. With many new or refactored components I started to use plain PHPUnit 9 and only resorted to Horde_Test when I strictly needed it. This simplified things a lot. The Horde organisation houses a dozen or so apps, almost 200 library repositories and some auxillary stuff. Having Horde_Test in the inheritance hierarchy makes any upgrades awkward. Need to be more strict about signatures for a new PHP release? Fix it all at once. Want to adopt a newer phpunit framework? Do it wholesale. Hook into some internal PHPUnit mechanism? They told us not to and they use their freedom to break internal classes very liberally. Tough luck.

So instead I am mostly abandoning the Horde_Test_Case base class and everything that comes with it.

  • PHPUnit hooks right into the composer autoloader. No more AllTests.php, autoload.php or bootstrap.php
  • I use the PHPUnit version I see fit. If I rarely update a component and it’s supposed to run on old PHP 7.4 through PHP 8.4, then I am fine when the tests work with PHPUnit 10.
  • If I develop something bleeding edge which does not need to work on PHP 8.2 and older, I simply start out at PHPUnit 12.
  • If I want to abstract out a test helper which is useful beyond the limits of an individual repo, I try to avoid building new bases classes. Instead I build it as a utility

Many libraries which get a wholesale upgrade no longer depend on Horde/Test. This is good because Horde/Test pulls in too many implicit dependencies anyway.

This comes at a price though. Our custom test runner relied on a specific structure of horde tests. The internal quality check pipeline in the horde-components tool also relied on a specific phpunit version and library layout. These are now broken and I won’t try that approach again.

So the new layout looks fairly standard.

  • Tests are in the /test/ subdirectory
  • Unit tests which are intended to run on most platforms are in /test/unit. Tests which require external resources, expensive calculations or rarely used PHP extensions get their own lowercase test suite directory
    /test/ldap/
    /test/pam/
    /test/kolab/
    /test/integration/
  • The horde-components tool autodetects these test suites and sets up the appropriate PSR-4 autoload-dev rules in the composer.json file
  • Just call “phpunit” to run the test suites.

bookmark_borderI want to run horde/components on new PHP

If you want to run horde-components on a version of PHP which is not yet reflected in packagist.org released versions:

me@mine:~/horde/components$ composer config minimum-stability dev
me@mine:~/horde/components$ composer install –ignore-platform-reqs

No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 90 installs, 0 updates, 0 removals

  • Locking fig/event-dispatcher-util (1.3.1)
  • Locking horde/alarm (v3.0.0alpha4)
  • Locking horde/argv (dev-FRAMEWORK_6_0 a1362b5)
  • Locking horde/auth (v3.0.0alpha6)
  • Locking horde/autoloader (dev-FRAMEWORK_6_0 dfb56fa)
  • Locking horde/browser (v3.0.0alpha4)
  • Locking horde/cache (dev-FRAMEWORK_6_0 29bb731)
  • Locking horde/cli (v3.0.0alpha5)
  • Locking horde/cli_modular (v3.0.0alpha4)
  • Locking horde/compress (v3.0.0alpha4)
  • Locking horde/compress_fast (v2.0.0alpha4)
  • Locking horde/constraint (v3.0.0alpha4)
  • Locking horde/controller (dev-FRAMEWORK_6_0 35005ea)
  • Locking horde/core (v3.0.0alpha14)
  • Locking horde/crypt_blowfish (v2.0.0alpha3)
  • Locking horde/css_parser (v2.0.0alpha4)
  • Locking horde/cssminify (v2.0.0alpha4)
  • Locking horde/data (v3.0.0alpha4)
  • Locking horde/date (v3.0.0alpha4)
  • Locking horde/dav (v2.0.0alpha4)
  • Locking horde/db (v3.0.0alpha4)
  • Locking horde/eventdispatcher (dev-FRAMEWORK_6_0 8253aa9)
  • Locking horde/exception (v3.0.0alpha4)
  • Locking horde/githubapiclient (dev-FRAMEWORK_6_0 fe17552)
  • Locking horde/group (v3.0.0alpha4)
  • Locking horde/hashtable (v2.0.0alpha4)
  • Locking horde/history (v3.0.0alpha4)
  • Locking horde/horde-installer-plugin (v2.5.5)
  • Locking horde/http (dev-FRAMEWORK_6_0 a98cb86)
  • Locking horde/http_server (dev-FRAMEWORK_6_0 4a91669)
  • Locking horde/icalendar (v3.0.0alpha4)
  • Locking horde/idna (v2.0.0alpha4)
  • Locking horde/injector (dev-FRAMEWORK_6_0 6fbc75e)
  • Locking horde/javascriptminify (v2.0.0alpha4)
  • Locking horde/listheaders (v2.0.0alpha4)
  • Locking horde/lock (v3.0.0alpha4)
  • Locking horde/log (v3.0.0alpha8)
  • Locking horde/logintasks (v3.0.0alpha4)
  • Locking horde/mail (v3.0.0alpha4)
  • Locking horde/mime (v3.0.0alpha5)
  • Locking horde/mime_viewer (v3.0.0alpha4)
  • Locking horde/nls (v3.0.0alpha4)
  • Locking horde/notification (v3.0.0alpha4)
  • Locking horde/pack (v2.0.0alpha4)
  • Locking horde/pear (dev-FRAMEWORK_6_0 6e99004)
  • Locking horde/perms (v3.0.0alpha4)
  • Locking horde/prefs (v3.0.0alpha6)
  • Locking horde/release (v4.0.0alpha4)
  • Locking horde/role (dev-FRAMEWORK_6_0 cac03aa)
  • Locking horde/rpc (v3.0.0alpha5)
  • Locking horde/secret (v3.0.0alpha4)
  • Locking horde/serialize (v3.0.0alpha4)
  • Locking horde/sessionhandler (v3.0.0alpha3)
  • Locking horde/share (v3.0.0alpha4)
  • Locking horde/stream (v2.0.0alpha4)
  • Locking horde/stream_filter (v3.0.0alpha4)
  • Locking horde/stream_wrapper (v3.0.0alpha4)
  • Locking horde/support (v3.0.0.1alpha4)
  • Locking horde/template (v3.0.0alpha4)
  • Locking horde/test (dev-FRAMEWORK_6_0 c512302)
  • Locking horde/text_diff (dev-FRAMEWORK_6_0 f7cfbc2)
  • Locking horde/text_filter (v3.0.0alpha3)
  • Locking horde/text_flowed (v3.0.0alpha4)
  • Locking horde/token (v3.0.0alpha4)
  • Locking horde/translation (v3.0.0alpha2)
  • Locking horde/url (v3.0.0alpha5)
  • Locking horde/util (dev-FRAMEWORK_6_0 f62a395)
  • Locking horde/view (v3.0.0alpha4)
  • Locking horde/xml_element (v3.0.0alpha4)
  • Locking horde/yaml (dev-FRAMEWORK_6_0 c222d58)
  • Locking pear/archive_tar (1.5.0)
  • Locking pear/console_getopt (dev-master f0098a8)
  • Locking pear/pear (dev-master 9d3ac5e)
  • Locking pear/structures_graph (dev-trunk 66368ac)
  • Locking pear/xml_util (dev-master a1ce442)
  • Locking php-extended/polyfill-php80-stringable (1.2.12)
  • Locking psr/container (dev-master 7079847)
  • Locking psr/event-dispatcher (dev-master bbd9eac)
  • Locking psr/http-client (dev-master bb5906e)
  • Locking psr/http-factory (1.1.0)
  • Locking psr/http-message (dev-master 402d35b)
  • Locking psr/http-server-handler (dev-master 13403d4)
  • Locking psr/http-server-middleware (dev-master 459eeb7)
  • Locking psr/log (dev-master f16e1d5)
  • Locking sabre/dav (4.7.x-dev 7183a67)
  • Locking sabre/event (5.1.x-dev 1538b1b)
  • Locking sabre/http (5.1.x-dev 4c2a2c0)
  • Locking sabre/uri (v2.x-dev 2c21ebd)
  • Locking sabre/vobject (4.5.x-dev ff22611)
  • Locking sabre/xml (2.2.x-dev 01a7927)
    Writing lock file
    Installing dependencies from lock file (including require-dev)
    Package operations: 90 installs, 0 updates, 0 removals
  • Downloading horde/horde-installer-plugin (v2.5.5)
  • Downloading horde/translation (v3.0.0alpha2)
  • Downloading horde/exception (v3.0.0alpha4)
  • Downloading horde/util (dev-FRAMEWORK_6_0 f62a395)
  • Downloading horde/nls (v3.0.0alpha4)
  • Downloading horde/date (v3.0.0alpha4)
  • Downloading horde/alarm (v3.0.0alpha4)
  • Downloading horde/auth (v3.0.0alpha6)
  • Downloading horde/autoloader (dev-FRAMEWORK_6_0 dfb56fa)
  • Downloading horde/compress_fast (v2.0.0alpha4)
  • Downloading horde/cache (dev-FRAMEWORK_6_0 29bb731)
  • Downloading horde/stream_wrapper (v3.0.0alpha4)
  • Downloading horde/support (v3.0.0.1alpha4)
  • Downloading horde/cli (v3.0.0alpha5)
  • Downloading horde/argv (dev-FRAMEWORK_6_0 a1362b5)
  • Downloading horde/cli_modular (v3.0.0alpha4)
  • Downloading psr/log (dev-master f16e1d5)
  • Downloading php-extended/polyfill-php80-stringable (1.2.12)
  • Downloading horde/constraint (v3.0.0alpha4)
  • Downloading horde/log (v3.0.0alpha8)
  • Downloading psr/container (dev-master 7079847)
  • Downloading horde/injector (dev-FRAMEWORK_6_0 6fbc75e)
  • Downloading horde/controller (dev-FRAMEWORK_6_0 35005ea)
  • Downloading horde/crypt_blowfish (v2.0.0alpha3)
  • Downloading horde/url (v3.0.0alpha5)
  • Downloading horde/css_parser (v2.0.0alpha4)
  • Downloading horde/cssminify (v2.0.0alpha4)
  • Downloading horde/text_flowed (v3.0.0alpha4)
  • Downloading horde/secret (v3.0.0alpha4)
  • Downloading horde/idna (v2.0.0alpha4)
  • Downloading horde/text_filter (v3.0.0alpha3)
  • Downloading horde/stream_filter (v3.0.0alpha4)
  • Downloading horde/stream (v2.0.0alpha4)
  • Downloading horde/mime (v3.0.0alpha5)
  • Downloading horde/mail (v3.0.0alpha4)
  • Downloading horde/listheaders (v2.0.0alpha4)
  • Downloading horde/icalendar (v3.0.0alpha4)
  • Downloading horde/browser (v3.0.0alpha4)
  • Downloading horde/data (v3.0.0alpha4)
  • Downloading psr/event-dispatcher (dev-master bbd9eac)
  • Downloading fig/event-dispatcher-util (1.3.1)
  • Downloading horde/eventdispatcher (dev-FRAMEWORK_6_0 8253aa9)
  • Downloading psr/http-message (dev-master 402d35b)
  • Downloading psr/http-factory (1.1.0)
  • Downloading psr/http-client (dev-master bb5906e)
  • Downloading horde/http (dev-FRAMEWORK_6_0 a98cb86)
  • Downloading horde/githubapiclient (dev-FRAMEWORK_6_0 fe17552)
  • Downloading horde/hashtable (v2.0.0alpha4)
  • Downloading horde/db (v3.0.0alpha4)
  • Downloading horde/history (v3.0.0alpha4)
  • Downloading psr/http-server-handler (dev-master 13403d4)
  • Downloading psr/http-server-middleware (dev-master 459eeb7)
  • Downloading horde/http_server (dev-FRAMEWORK_6_0 4a91669)
  • Downloading horde/javascriptminify (v2.0.0alpha4)
  • Downloading horde/lock (v3.0.0alpha4)
  • Downloading horde/logintasks (v3.0.0alpha4)
  • Downloading horde/compress (v3.0.0alpha4)
  • Downloading horde/mime_viewer (v3.0.0alpha4)
  • Downloading horde/notification (v3.0.0alpha4)
  • Downloading horde/pack (v2.0.0alpha4)
  • Downloading horde/yaml (dev-FRAMEWORK_6_0 c222d58)
  • Downloading horde/xml_element (v3.0.0alpha4)
  • Downloading horde/pear (dev-FRAMEWORK_6_0 6e99004)
  • Downloading horde/prefs (v3.0.0alpha6)
  • Downloading horde/serialize (v3.0.0alpha4)
  • Downloading horde/group (v3.0.0alpha4)
  • Downloading horde/perms (v3.0.0alpha4)
  • Downloading sabre/uri (v2.x-dev 2c21ebd)
  • Downloading sabre/xml (2.2.x-dev 01a7927)
  • Downloading sabre/vobject (4.5.x-dev ff22611)
  • Downloading sabre/event (5.1.x-dev 1538b1b)
  • Downloading sabre/http (5.1.x-dev 4c2a2c0)
  • Downloading sabre/dav (4.7.x-dev 7183a67)
  • Downloading pear/pear (dev-master 9d3ac5e)
  • Downloading pear/xml_util (dev-master a1ce442)
  • Downloading pear/structures_graph (dev-trunk 66368ac)
  • Downloading pear/console_getopt (dev-master f0098a8)
  • Downloading pear/archive_tar (1.5.0)
  • Downloading horde/view (v3.0.0alpha4)
  • Downloading horde/token (v3.0.0alpha4)
  • Downloading horde/template (v3.0.0alpha4)
  • Downloading horde/share (v3.0.0alpha4)
  • Downloading horde/sessionhandler (v3.0.0alpha3)
  • Downloading horde/core (v3.0.0alpha14)
  • Downloading horde/dav (v2.0.0alpha4)
  • Downloading horde/rpc (v3.0.0alpha5)
  • Downloading horde/release (v4.0.0alpha4)
  • Downloading horde/role (dev-FRAMEWORK_6_0 cac03aa)
  • Downloading horde/test (dev-FRAMEWORK_6_0 c512302)
  • Downloading horde/text_diff (dev-FRAMEWORK_6_0 f7cfbc2)
  • Installing horde/horde-installer-plugin (v2.5.5): Extracting archive
  • Installing horde/translation (v3.0.0alpha2): Extracting archive
  • Installing horde/exception (v3.0.0alpha4): Extracting archive
  • Installing horde/util (dev-FRAMEWORK_6_0 f62a395): Extracting archive
  • Installing horde/nls (v3.0.0alpha4): Extracting archive
  • Installing horde/date (v3.0.0alpha4): Extracting archive
  • Installing horde/alarm (v3.0.0alpha4): Extracting archive
  • Installing horde/auth (v3.0.0alpha6): Extracting archive
  • Installing horde/autoloader (dev-FRAMEWORK_6_0 dfb56fa): Extracting archive
  • Installing horde/compress_fast (v2.0.0alpha4): Extracting archive
  • Installing horde/cache (dev-FRAMEWORK_6_0 29bb731): Extracting archive
  • Installing horde/stream_wrapper (v3.0.0alpha4): Extracting archive
  • Installing horde/support (v3.0.0.1alpha4): Extracting archive
  • Installing horde/cli (v3.0.0alpha5): Extracting archive
  • Installing horde/argv (dev-FRAMEWORK_6_0 a1362b5): Extracting archive
  • Installing horde/cli_modular (v3.0.0alpha4): Extracting archive
  • Installing psr/log (dev-master f16e1d5): Extracting archive
  • Installing php-extended/polyfill-php80-stringable (1.2.12): Extracting archive
  • Installing horde/constraint (v3.0.0alpha4): Extracting archive
  • Installing horde/log (v3.0.0alpha8): Extracting archive
  • Installing psr/container (dev-master 7079847): Extracting archive
  • Installing horde/injector (dev-FRAMEWORK_6_0 6fbc75e): Extracting archive
  • Installing horde/controller (dev-FRAMEWORK_6_0 35005ea): Extracting archive
  • Installing horde/crypt_blowfish (v2.0.0alpha3): Extracting archive
  • Installing horde/url (v3.0.0alpha5): Extracting archive
  • Installing horde/css_parser (v2.0.0alpha4): Extracting archive
  • Installing horde/cssminify (v2.0.0alpha4): Extracting archive
  • Installing horde/text_flowed (v3.0.0alpha4): Extracting archive
  • Installing horde/secret (v3.0.0alpha4): Extracting archive
  • Installing horde/idna (v2.0.0alpha4): Extracting archive
  • Installing horde/text_filter (v3.0.0alpha3): Extracting archive
  • Installing horde/stream_filter (v3.0.0alpha4): Extracting archive
  • Installing horde/stream (v2.0.0alpha4): Extracting archive
  • Installing horde/mime (v3.0.0alpha5): Extracting archive
  • Installing horde/mail (v3.0.0alpha4): Extracting archive
  • Installing horde/listheaders (v2.0.0alpha4): Extracting archive
  • Installing horde/icalendar (v3.0.0alpha4): Extracting archive
  • Installing horde/browser (v3.0.0alpha4): Extracting archive
  • Installing horde/data (v3.0.0alpha4): Extracting archive
  • Installing psr/event-dispatcher (dev-master bbd9eac): Extracting archive
  • Installing fig/event-dispatcher-util (1.3.1): Extracting archive
  • Installing horde/eventdispatcher (dev-FRAMEWORK_6_0 8253aa9): Extracting archive
  • Installing psr/http-message (dev-master 402d35b): Extracting archive
  • Installing psr/http-factory (1.1.0): Extracting archive
  • Installing psr/http-client (dev-master bb5906e): Extracting archive
  • Installing horde/http (dev-FRAMEWORK_6_0 a98cb86): Extracting archive
  • Installing horde/githubapiclient (dev-FRAMEWORK_6_0 fe17552): Extracting archive
  • Installing horde/hashtable (v2.0.0alpha4): Extracting archive
  • Installing horde/db (v3.0.0alpha4): Extracting archive
  • Installing horde/history (v3.0.0alpha4): Extracting archive
  • Installing psr/http-server-handler (dev-master 13403d4): Extracting archive
  • Installing psr/http-server-middleware (dev-master 459eeb7): Extracting archive
  • Installing horde/http_server (dev-FRAMEWORK_6_0 4a91669): Extracting archive
  • Installing horde/javascriptminify (v2.0.0alpha4): Extracting archive
  • Installing horde/lock (v3.0.0alpha4): Extracting archive
  • Installing horde/logintasks (v3.0.0alpha4): Extracting archive
  • Installing horde/compress (v3.0.0alpha4): Extracting archive
  • Installing horde/mime_viewer (v3.0.0alpha4): Extracting archive
  • Installing horde/notification (v3.0.0alpha4): Extracting archive
  • Installing horde/pack (v2.0.0alpha4): Extracting archive
  • Installing horde/yaml (dev-FRAMEWORK_6_0 c222d58): Extracting archive
  • Installing horde/xml_element (v3.0.0alpha4): Extracting archive
  • Installing horde/pear (dev-FRAMEWORK_6_0 6e99004): Extracting archive
  • Installing horde/prefs (v3.0.0alpha6): Extracting archive
  • Installing horde/serialize (v3.0.0alpha4): Extracting archive
  • Installing horde/group (v3.0.0alpha4): Extracting archive
  • Installing horde/perms (v3.0.0alpha4): Extracting archive
  • Installing sabre/uri (v2.x-dev 2c21ebd): Extracting archive
  • Installing sabre/xml (2.2.x-dev 01a7927): Extracting archive
  • Installing sabre/vobject (4.5.x-dev ff22611): Extracting archive
  • Installing sabre/event (5.1.x-dev 1538b1b): Extracting archive
  • Installing sabre/http (5.1.x-dev 4c2a2c0): Extracting archive
  • Installing sabre/dav (4.7.x-dev 7183a67): Extracting archive
  • Installing pear/pear (dev-master 9d3ac5e): Extracting archive
  • Installing pear/xml_util (dev-master a1ce442): Extracting archive
  • Installing pear/structures_graph (dev-trunk 66368ac): Extracting archive
  • Installing pear/console_getopt (dev-master f0098a8): Extracting archive
  • Installing pear/archive_tar (1.5.0): Extracting archive
  • Installing horde/view (v3.0.0alpha4): Extracting archive
  • Installing horde/token (v3.0.0alpha4): Extracting archive
  • Installing horde/template (v3.0.0alpha4): Extracting archive
  • Installing horde/share (v3.0.0alpha4): Extracting archive
  • Installing horde/sessionhandler (v3.0.0alpha3): Extracting archive
  • Installing horde/core (v3.0.0alpha14): Extracting archive
  • Installing horde/dav (v2.0.0alpha4): Extracting archive
  • Installing horde/rpc (v3.0.0alpha5): Extracting archive
  • Installing horde/release (v4.0.0alpha4): Extracting archive
  • Installing horde/role (dev-FRAMEWORK_6_0 cac03aa): Extracting archive
  • Installing horde/test (dev-FRAMEWORK_6_0 c512302): Extracting archive
  • Installing horde/text_diff (dev-FRAMEWORK_6_0 f7cfbc2): Extracting archive
    73 package suggestions were added by new dependencies, use composer suggest to see details.
    Package php-extended/polyfill-php80-stringable is abandoned, you should avoid using it. Use php >= 8.0 instead.
    Generating autoload files
    Applying /presets for absent files in /var/config
    Looking for registry snippets from apps
    Writing app configs to /var/config dir
    Linking app configs to /web Dir
    Linking javascript tree to /web/js
    Linking themes tree to /web/themes

Beware, these constraints are there for a reason. Expect things to break in unexpected ways if versions are actually incompatible.

bookmark_borderPrivilege Separation: Two github logins on the same Linux or WSL

Scenario:
You run a linux or wsl-equipped windows development machine where you do 90% of work for organization A using github.com and 10% for “other” and you strictly must not use the same github account. Both types of work require multiple repos so handling access per-repo is tedious.

Scenario B: Privilege separation

You have an account with high privilege on certain repos which you need to keep, i.e. to override CI failure on time critical issues, be able to block access on short notice or do things you would not allow the juniors to do. But you want to saveguard your 95% daily work against accidentally doing something to the wrong repo

Approach:
Have two separate operating system users. Actual access control on the filesystem is not the issue we want to tackle but separation of accounts. You can even “share” the code

mkdir /srv/develop
chown -R primaryuser:users /srv/develop
chmod chmod g+rwx /srv/develop

Now login every user to his appropriate github account

sudo su – primaryuser
ln -s /srv/develop /home/primaryuser/develop
gh auth login -h github.com -w -phttps
gh auth login corporategithub.com -w -phttps

Don’t forget to logout and login to the other ui user

sudo su – otheruser
ln -s /srv/develop /home/otheruser/develop
gh auth login -h github.com -w -phttps
gh auth login -h othercorporategithub.com -w -phttps

Logout again. After this point you don’t need to login to different browser sessions all the time.

You can also use prepared github personal tokens of each users and save the web browser hassle. I chose to go the UI way this time.

You can freely connect IDE’s to repos using your primary account for both types of repos.
The only thing you must avoid is pushing and PRing through the IDE.

Instead have a terminal window for each type of account and do it there

git push —
# This should not be possible to get wrong. Github will not allow you to push to a repo the account does not have access to.

gh pr create –fill
# Create PRs without hitting the browser and avoid all the login handling. Alternatively you can use another browser profile or browser install, i.e. use Firefox for your oddball account and your primary browser (probably something chromium based) for the main use case.

bookmark_borderRunaway Trains Back on Track – Plans are just plans

Making plans is great but it’s just words. How are things going? Clearly moving towards goals but in unexpected ways.

Trains don’t only run on railroad tracks but supposedly on schedule. Commuters shall know every stop, arrival and departure, the order of cars and their position on the platform. If trains regularly arrive delayed, at another platform, composed of different or differently ordered cars it is a source of frustration. If they skip a station or get relayed to another course, it’s even worse. Software projects, on the other hand, are expected to detour regularly in very similar ways but hopefully arrive at certain way points more or less on time. Sometimes they get unplanned additional cars along the way. Sometimes you have to replace the engine. It never gets boring.

What is the goal?

In January, I anticipated there were some unknowns ahead but I did not detail how to tackle this.

Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial. 

Back to the source code? The 2025 agenda – ralf-lang.de

The Märklin Hardware Throttles (Mobile Station 2) implement hot plugging and negotiate which one acts as the primary. They send a regular PING command announcing their type and serial and expect a PONG response from other devices on the bus. The most senior serial number is agreed to be the primary throttle even if the other one was the primary or single throttle until now. The primary offers its roster to the other throttles. If higher end Central Station systems are on the bus, one of the central station will always be the primary device and the throttles all behave a little different. This allows JMRI to passively listen to traffic and detect bus devices or actively send a PING and collect the PONGs.

The Can-Digital-Bahn devices have a similar PING/PONG mechanism used by their configuration utilities. Unfortunately there are different configuration utilities for different generations of their sensors, relays, turnout controllers, light controllers, switchboard components, mixed devices. These utilities are closed source Windows programs and you need different versions of the program for different versions of the components. Unfortunately, not all of the programs properly run on recent Windows systems.

What if I just had to push a button in the web browser and see all my MCAN throttles and devices in a table, loading the right configuration screen for each of them? Sounds much better, but how to achieve that?

Moving through the stack

To send the PING messages and receive the PONG messages a program needs to connect to the MCAN bus. This can be achieved by connecting the CC-Schnitte interface to the USB port of the computer or connecting to a Central Station device over TCP/IP. Both methods expose the raw MCAN bus to the software. Both options are already implemented in JMRI.

Next the necessary bus messages and response need to be added to the software. The MCAN PING command is already implemented but JMRI doesn’t implement the desired reactions to the answers. For can-digitial-bahn messages, support is still missing.

To use the found devices, a program needs to memorize some representation of them. JMRI currently does not have a generic way of handling external bus devices as such. Throttles and sensors are tracked in their specific roles, covering their common aspects as throttles or as sensors. For CBUS type connections, there is a node manager which looks very similar to what is needed for MCAN but the UI code and the table format probably deviates a bit. This is where it becomes a bit complicated.

Users need some kind of GUI to interact with found devices. There are many reasons to start out with a browser based UI instead of building it in JMRI’s native Java GUI. The browser based ui just works on any tablet or laptop connected to the network. It’s well decoupled from JMRI’s core. It can be styled or completely recomposed as needed without touching JMRI itself. JMRI does not need to bundle variants for different use cases but they can be installed separately, developed on their own schedule. JMRI only needs to expose the necessary interface protocol which also drives other parts of the browser based UI.

JMRI’s interface protocol is composed from JSON messages between the browser client and the JMRI server. These can be sent over the HTTP server port or through a separate WebSocket implementation. The latter provides better performance and less overhead.

JMRI’s WebSockets and HTTPS API

This is what I ended up doing. To familiarize myself with the API I built a simple demo use case to manipulate the displayed railroad company name. Previously this was only possible through the preferences screen in the native Java GUI. I ended up not integrating it in the regular browser based GUI but implement a separate demo page. JMRI core developers pointed out that special care is needed and unauthenticated calls to the API must not actually persist the changed configuration into JMRI.

Obviously this would not work for actually handling hardware. While JMRI already has a permissions and user authentication system, it is fairly new and does not cover the actual API messages. So my next step would be to implement the necessary messages in the JSON WebSocket API for retrieving an authentication token. This token would then be used to authorize further WebSocket calls which actually change something in JMRI. These new calls need to carry the authorization token as part of the data.

I plan to detail this development in a separate article later this month as I move through the process.

Seems like I should be getting somewhere

This free time project is a great adventure into the unknown between where I am and where I want to be next. In some ways this is similar to professional work. I commit to goals and target dates and I have a very clear understanding of next week and some ideas about the week after. Beyond that the way points and schedule dates become rarer.

It’s sometimes very challenging to tell project managers that there won’t be many super detailed milestones beyond the horizon of immediate next steps. Gladly, at work I have very smart project managers who know when to trust me and when to challenge me. Sadly, in free time I have a very harsh and unforgiving project manager who often won’t take no for an answer.

But that’s just me.

bookmark_borderYears after cancellation: New Patch for Imperator

Paradox Development Studios is famous for their grand strategy games including the flagship Europa Universalis franchise. While a new main version is developed under code name Project Caesar, the latest Europa Universalis IV still gets new expansion packs (Download Content) and updates to the base game. Paradox even backported several former add-ons into the latest version of the base games. We are talking about a product originally released in 2013, almost a teenager now.

Their 2019 spin off Imperator: Rome turned out to be a much smaller commercial success. After a few add-ons and updates it got cancelled in 2021 due to dwindling sales. It was not their first foray into ancient history. Europa Universalis: Rome debuted in 2008 but it was not a long lived installment. Only one download content got produced before the product was relegated to the back catalog.

While Imperator: Rome is based on technology introduced for Europa Universalis 4 and the concurrent generation of spinoffs, Europa Universalis: Rome is clearly a member of the Europa Universalis 3 generation of the engine, UI design and scripting approaches. Imperator debuted the Jomini middleware between the Clausewitz engine and the actual game content. Jomini has since been used in later installments of various Paradox Grand Strategy games.

Even though Imperator product development has been cancelled in 2021, Paradox employees have still worked on bug fixes and feature patches. In April 2024, a new version 2.0.4 has been released for all supported platforms after it was available as an opt-in beta version for several months. In December 2024 a new patch 2.0.5 has been made available as public beta. The patch contains minor UI tweaks, some fixes to the computer player AI, some bug fixes and many additional hooks and exposed variables for the scripting engine. This allows builders of unofficial game modifications (“Mods”) to add new behaviours to the game like Trade Embargoes or forbidding some game characters to inherit provinces. Imperator: Invictus is the most prominent modification for the official Imperator game. It works both with the latest version of the base game and optionally all available official addons.

I find it very motivating and inspiring when a vendor continues to work with fans and open source community and supports their efforts to add more value to their discontinued commercial product.

As of January 2025 Imperator: Rome and all individual add-ons are available at a heavy discount from Paradox’ own store. At the moment they offer the individual components for less money than the official bundle. I am in no way affiliated with Paradox but I like the way they do this and the game is really fun.

bookmark_borderBack to the source code? The 2025 agenda

Looking back on 2024, it was a challenging, demanding and sometimes crazy year. But let’s start with athe good things in life: There was so much joy to be had with the family, so much to learn and discover at work, so much progress on personal development and discovering new approaches to things that just won’t go away. All these precious moments I would not want to miss. It would be wrong to just skip over this without this little line of appreciation. I benefit very much from my family, friends and colleagues and I hope to give back a little bit of joy and a helping hand wherever I can. Thank you all.

But if you know me: I am ambitious, I am loyal to stuff I started, sometimes stubborn and I am an open source software guy.
From this perspective, things were not going so well. While I am used to full schedules and stressful timelines at work, I had to re-prioritize my free time harshly in the middle of the first quarter. This got a lot of private projects into an undeclared hiatus and ultimately frozen for the rest of the year. It came unexpected and initially I did not grasp the extent of what was happening. To give you an idea, I even stopped reading mailing lists and most of my private inbox for a while. I did not plan any public talks and only visited FOSS events if they were job related. By coincidence, the main laptop with open source work setups also went defunct and I did not bother to replace it for several months.

If you feel you get into a similar situation, I can only advise you to take no time for regrets. Do what needs to be done and cut what you’d like to do on top until you feel ready again. Watch for the metrics: Do you start reading again? Is it more pages of fiction and non-fiction than last month? Do you take time for the occassional board or computer game? Can you make time again for you hobby, club or sports activities? Then maybe you’re set to come back to free time FOSS development – if you still like it. Don’t feel you need or must or have to. Maybe take another two weeks to question yourself before you start over. Nothing is won with spending hours in an exhausted or half detracted state.

That said, I am very positive to slowly bring everything back to a more happy status one at a time. A new FOSS work setup is available. Some of 2024’s learnings may help me to achieve more while still on a limited time budget. I am catching up with mails and yes, your pull requests will be handled. There’s also some clerical work to be done for my pet projects. It’s too early to announce a specific timeline or commitment and sometimes opportunity trumps priority. Let’s see how that works.

So what’s on the agenda?

  • Horde 6 Framework and main apps need to go to “stable”. I don’t want any ambiguity anymore. Make it run on the recent and upcoming PHP and try not to break it for any officially maintained release +1
  • PoC browser & thunderbird plugin with some very limited initial use case. Maybe a sidebar for Horde Mnemo Notes. Let’s gather some experience before making big plans.
  • Quality of life for aspiring developers: Documentation, test setup, reference setup
  • Modern authentication focus: I’ve done a custom 2FA/TOTP solution several years ago and I have been meaning to port it to Horde’s core for several years. The authentication stack probably needs to be revisited to make it a natural fit. I’d like to keep it nicely split into an interface part and the actual implementation so exchanging the TOTP app for some QR Code Challenge on your device, email confirmation or similar should be possible without modifying the core code.
  • Continue porting some traditional pages to the PSR-7 standard based controller framework
  • Be more active again in PHP-FIG discussions and practical maintenance work on the PSR/PER related repositories
  • Having loads of fun and learning
  • Improve JMRI‘s support for the Märklin MCAN protocol and particularly the Mobile Station 2, Central Station 3 and Can-Digital-Bahn products. This might become a bit controversial. The current implementation is very much centered on fulfilling JMRI’s minimal requirements for a connection and throttle implementation and much code is a straight port from the TAMS implementation. I’d rather build a java library with internal cohesion and a good match to the bus protocol and the device philosophy. I have read into Bob’s more specialized TrainControl and Frans Jacob’s standalone program. I appreciate how much more cohesive and maintainable their implementation is. They achieve this by not letting an external frame dictate their internal structure and mental model. Of course this will need some glue to match it with JMRI’s wider logic but it’s also a great opportunity to simplify unit testing.

That’s already a tall order and I am sure more ideas and desires pop up along the way. No explicit priority, no promises, no commitment.

bookmark_borderRun FTDI USB devices on Windows 10/11 WSL2

Issue: You can use FTDI based USB devices such as microcontroller boards and embedded products in Ubuntu Linux on physical devices and on full flegded VMs but under WSL2 the device does not work. You can successfully map the USB device in usbipd and you see it show up in lsusb, dmesg, journal. But no new serial device shows up under /dev/tty* and you cannot use the device.

Background: Ubuntu under Windows 11 WSL2 as of 2024/04 does not include the ftdi-sio driver in its default kernel builds. You cannot easily install that driver from the package manager either. Under WSL, similar to container environments and solaris zones, all linux guests use the same kernel.

Solution: Build and install a custom linux kernel with builtin FTDI drivers. At the time of writing, the default WSL kernel is 5.15 and the latest WSL branch is 6.1

Enter the Ubuntu WSL environment:

# If ubuntu is your only WSL or your default WSL, simply type "wsl"
# Double check with wsl -l if your ubuntu has another name like Ubuntu-22.04
# Mine is just called "Ubuntu"
wsl -d Ubuntu

Now download Microsoft’s modified kernel distribution and build it

git clone https://github.com/microsoft/WSL2-Linux-Kernel.git --depth=1 -b linux-msft-wsl-6.1.y
sudo apt update
sudo apt -y install build-essential flex bison libssl-dev libelf-dev bc 
sudo apt -y install python3 pahole
cd WSL2-Linux-Kernel
## This line turns the FTDI driver from disabled or module to builtin
./scripts/config --file Microsoft/config-wsl --set-val CONFIG_USB_SERIAL_FTDI_SIO y
make -j$(nproc) KCONFIG_CONFIG=Microsoft/config-wsl
sudo make modules_install headers_install
## Use another target directory if you don't want the kernel in the root of C:\
cp arch/x86/boot/bzImage /mnt/c/bzImage-6.1

Leave the VM.
Back in Windows Terminal, run

wsl --shutdown
cd ~
notepad .wslconfig

Add or edit in the file:

[wsl2]
kernel=C:\\Users\\yourusernamehere\\bzImage-6.1
memory=4GB

Save the file.
Then run “wsl” or “wsl -d yourdistribution” to restart into the new kernel
To verify inside wsl, run

uname -a
## Expect output similar to
## Linux W-PF384K2W 6.1.21.2-microsoft-standard-WSL2+ #1 SMP Tue Apr  2 22:11:36 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux

Note: In some cases, Windows will not actually terminate WSL and you need a full reboot of the host windows computer.