How Prosody developers spent 2020

Nobody here knew quite what a year 2020 was going to be! However despite pandemics and lockdowns, we have continued to work on Prosody. This post is a summary of how the project is doing, and what we’ve been up to in the past year.

One quick note before we begin… Prosody is an independent open-source project and exists only because the developers have been fortunate enough to be in a position to work on it. A couple of core team members are currently looking for freelance work. If you have projects in need of a Prosody expert, check the bottom of this post for more details!

More users than ever before

Prosody does not “phone home” in any way, which means we do not have a lot of insight into how many people are using Prosody. But there are some indicators that we can use to see at least the growth of the project.

Many years ago, circa 2014, I was filling out a form that asked how many users the project had. I thought long and hard, but with no idea how to measure, I wrote down an estimate of “500” based on nothing but a gut feeling. Only a few weeks later I learned that the XMPP Observatory had already seen over 2200 domains submitted that were running Prosody. As most deployments were unlikely to have been submitted to xmpp.net, my estimate was clearly far out. These days I jump at any chance for even a vague estimate of our userbase. It helps us to know that people are out there!

One useful tool is Shodan. This project scans the entire internet, just to see what it can find, and it records the results. Often used by academics and security researchers, a free account can also be used by anyone to run simple queries over the data they collect.

A Shodan search in 2017 turned up nearly 7000 Prosody deployments. The same search 8 months later returned over 16000. Today we’re at over 52000 Prosody servers! And this only counts instances using port 5269 and accessible to the internet. There are also many private/internal deployments of Prosody that are not included in these numbers. Unfortunately we didn’t run a report in 2019, but here’s a graph of the previous reports we have run:

Shodan reports over 85000 federating XMPP servers on port 5269 in total. Based on this, Prosody makes up 44% of the public XMPP network. That’s quite an achievement!

Another handy insight into one sector of Prosody deployments is via Debian’s “popularity contest” service. This is an automated survey that administrators of Debian servers can choose to opt into. It reports anonymously to Debian what software packages are installed and in use. Although it reflects only a small slice of even Debian installations, it is useful to see trends.

March 2020 marked an unprecedented spike in Prosody installations!

Although we don’t know for certain, we suspect this was caused by an surge of interest in the self-hosted video conferencing software, Jitsi Meet. Jitsi Meet integrates with Prosody, which is used to power the authentication, signaling and chat of the video conferences. Jitsi’s Videobridge component handles the media routing. Together they make a very powerful and flexible communication system, and it is hardly surprising that interest has spiked this year when there has been a massive shift to remote work, and online meetings have replaced physical ones.

The code counts

Now let’s look at some stats about the Prosody codebase itself.

Language	Files	Code	Comment	Comment %	Blank	Total
Lua	295	48358	3536	6.8%	6483	58377
C	13	2613	346	11.7%	643	3602
Other (build tools, etc.)	11	1551	22	6.3%	138	1711
————————-	——	———-	———-	———-	———-	———
Total	323	54760	3909	6.7%	7265	65934

Changes in 2020

In 2020 (looking at the development branch) we added 597 commits, changing 191 files. The changes added 9872 lines of code, and 3637 lines of code were removed.

A significant portion of the new lines were in our unit and integration tests (2200 lines, about 22%) which we have been working hard to expand over the past couple of years.

Community modules

If you use Prosody, you know we have an emphasis on modularity and extensibility. We like to make developing plugins for Prosody as easy as possible, whether it’s for integration with other systems or crazy experiments.

Here’s a random selection of the 363 modules currently in the repository:

mod_muc_eventsource

Receive messages from MUC rooms with 4 lines of Javascript

mod_log_ringbuffer

Send debug logs to RAM unless they are needed.

mod_component_client

Allow a Prosody server to run as a (XEP-0114) external component of another (Prosody or not) XMPP server.

mod_firewall

Powerful rules-based scripts for filtering and redirecting XMPP traffic.

mod_minimix

An experiment in making MUC joins persistent (like MIX).

During 2020 we saw 37 new modules published in the community repository, and 499 commits from 19 contributors. Together they added over 10,000 lines of code (and removed 728 lines). This makes it our most active year apart from 2018!

Outside of the Prosody dev team, the most active contributor was Seve, who also made their first contribution this year and added a total of four new modules. Welcome aboard!

Features

But the most important part… what features have we been working on? All these things are scheduled for the 0.12 release (more on that in a bit).

Plugin installer

We have been applying further polish to, and setting up the infrastructure for the plugin installer. This was a Google Summer of Code project by João Duarte. It utilizes the Lua package manager, LuaRocks, to download, install and manage community modules.

Although the the installer was completed in 2019, to make it generally usable we also had to ensure every module in the community repository could be packaged and served in an automated way by our server. We now have this working.

Bye bye telnet, hello prosodyctl shell!

The telnet console is one of the best things about Prosody, and we’ve been working on its successor. An early version of prosodyctl shell is already available to try out in trunk nightly builds.

Using prosodyctl allows us to more easily support advanced features such as line editing and history (previously attainable using a third-party utility, rlwrap). It also allows for some richer UIs and is more secure on shared servers (it uses a unix socket instead of TCP).

DNS improvements

Since Prosody needs to resolve special DNS record types (such as SRV records) and in an asynchronous manner, the built-in operating system APIs are generally inadequate.

For a long time we’ve been using an adopted library simply known as ‘dns.lua’ combined with our own asynchronous wrapper around it. Although it hasn’t been terrible, it has a few issues, especially in some uncommon environments. It also doesn’t support many advanced features such as DNSSEC.

Now we are migrating to libunbound, part of the unbound project. This is one of the leading DNS implementations, and will be a big improvement over our current DNS library. To try it out, you can simply install luaunbound (already available in luarocks, Debian testing, AUR and others - poke your distro maintainers if you don’t have it yet!).

HTTP server upload performance

We didn’t set out to write a HTTP server, but we ended up with one anyway! Originally added so that we could natively support BOSH (XMPP over HTTP) clients, it grew to support websockets, and various modules now provide HTTP APIs for integration between Prosody and other systems.

One big problem is that the original implementation was designed for only small amounts of data. Since the widespread of adoption of XEP-0363 people now want to be able to upload files, pictures and videos using Prosody’s internal HTTP server. We have limits in place to protect against denial of service attacks, but those same limits prevent large uploads from trusted users.

We’ve put some work into supporting “streaming uploads”, where incoming data can be saved directly to disk instead of RAM. This means it will be safe to increase file upload limits without opening up your server to increase RAM usage and denial of service attacks.

In general though, we do recommend using a real external HTTP server in a production or high traffic deployment (using mod_http_upload_external).

Beyond passwords

Passwords are the fundamental means of authenticating to your server in XMPP today. XMPP is quite good at this, adopting strong standard authentication mechanisms such as SCRAM far earlier than the rest of the industry. But the rest of the industry is also moving away from passwords in many places. We’re aiming to follow this movement also. Not that we are scrapping passwords entirely, but making it easier to offer alternatives.

Prosody actually has a number of non-password authentication modules already, such as mod_auth_oauthbearer (OAuth2 tokens), mod_auth_ccert (client certificates) and mod_auth_token (HMAC-based tokens). But most of the modules have limitations and are not well integrated (e.g. you can set up Prosody to accept passwords, or set it up to accept tokens, but you can’t offer both methods at the same time).

An important related aspect is authorization. In most systems authentication via a token also provides limited access to the account (e.g. if a password is associated with an account, a session that logged in using a token should not be allowed to reset the password).

We’ve been working on two things. Firstly, a built-in authorization system (more flexible than the current admins configuration option) where users and sessions can be associated with specific permissions and roles.

Secondly we’re using this authorization layer to add built-in support for OAuth2-style authentication and authorization.

This is exciting for a number of reasons. It will allow, for example, specialized clients to request and receive (when granted by the user) limited access to an account without giving it your password. Some example scenarios could be:

• Allow a third-party service to send messages via your account, but not access your contacts or message history
• Allow an account backup/migration tool read-only access to your account without giving it your password
• A specialized client could be granted access to only specific types of incoming messages. For example a chess game that used your XMPP account to communicate with a remote player, but wouldn’t be able to read your IM messages.

We’re still working out the details right now, but the expectation is that some of these features may require or be enhanced by extensions to the XMPP protocol. Once we have some implementation experience, we will standardize such extensions as XEPs. That way we can open the door to a world where every XMPP client doesn’t require a password, and doesn’t automatically get full access to your account.

The boring branches

All the above work has been happening in our trunk branch (nightly builds are available, by the way! ). But we are also maintaining our 0.11 stable branch with bug fixes.

In 2020 we made 98 commits to the 0.11 branch, and made 4 releases. The latest version is 0.11.7, and 0.11.8 will be coming soon.

We also have two other branches open - 0.9 (old old stable) and 0.10 (old stable). We have a soft policy to keep supporting our branches while they are in an active Debian release. Debian stable is on 0.11.x, and the previous Debian release (stretch) was on 0.9.x and stopped receiving security updates from the core Debian team in July. As such, we will be formally deprecating these branches shortly.

In summary: if you are not on 0.11.x already, get moving :)

Not to be missed…

Through 2020 we put a lot of attention into helping increase adoption of XMPP and helping it reach new users. This includes the launch of our sister project, Snikket at FOSDEM, and the backporting of Snikket’s invite-based registration to generic Prosody deployments.

Up next: Prosody 0.12

We’re currently working towards the next major release of Prosody. It’s a lot of work to polish off all the new features and get them sufficiently tested and documented. We are likely to start with a beta or release candidate in the coming months.

Many thanks to all the testers of the Prosody trunk nightly builds, who provide valuable feedback and deployment experience.

Available for hire

Prosody has been in active development for over a decade. Over this time we’ve deliberately strived to keep the core project free from any commercial influence or activities. We believe that this helps keep the project focused on what the community wants, instead of simply what people with money want :)

That said, we also need to pay our bills so we can continue to work on Prosody. Two members of the dev team are currently available for freelancing on a full or part-time basis for any size project. We do strongly prefer projects related to Prosody/XMPP, and open-source if possible.

Whether you want to hire us to integrate Prosody with your application, review your architecture, or develop a new feature in Prosody itself, get in touch. You can reach us at developers@prosody.im.

Finally, we hope everyone is having a great start to the new year. That’s all for now. Stay healthy!

---