The goal of this project was to port matrix-ircd from the outdated combinators style of futures 0.1 to the new-and-improved style of futures 0.3. As initially proposed, this greatly improves code readability and removes the convoluted compiler errors that were pervasive in futures 0.1
matrix-ircd
The matrix-ircd project functions as a bridge between two chat platforms: Internet relay chat (irc) and Matrix. matrix-ircd lets you use any standard IRC Client to communicate with Matrix chatrooms and direct messages.
Mentors
This project is mentored by Philipp Mandler (@phlmn) and Jonas Platte (@jplatte). I would like to thank them for the helpful advice and code reviews they have given me over the course of GSoC 2020.
Project Results
Since much of the code was written in late 2016, there were many portions of the code that were unidiomatic and produced compiler errors. The first step I made was to remove all compiler errors from cargo and clippy in #64.
Additional tests were then written in various parts of the codebase to ensure that the upgrade would produce the same results. These changes were made in #64 and #66.
Since the project master branch currently used a custom http implementation, I moved it to use hyper, a fast and correct http implementation that already supported async/await. Additionally, I moved the module that most utilized http, matrix, to async-await and the new hyper code. Not only did these changes shrink the code by ~700 lines, they also removed a lot of unnecessary complexity. These changes were made in #67.
The last large module remaining, irc, was then ported to irc, along with its dependencies in stream_fold.rs and its upstream user in the bridge module. The most exciting part of these changes was the removal of the futures 0.1 dependency. All code was now running in futures 0.3! These changes also moved away from tasked-futures which further improved code readability. This PR was in #71.
Since the bulk of the changes were now complete, I moved onto bug fixes. As @jplatte mentioned in #71, the current single threaded approach to the application was not ideal. In #72 I updated all code to be multithreaded-compatible.
Lastly, In #77 I included more tests to functions with heavy changes, added additional logging, removed unnecessary complexity that was introduced to keep code as "1:1" as possible, and added TLS support to hyper so that https would work. This patch also fixed a rather difficult bug regarding the irc TCP streams and the buffer they were reading into.
Remaining Work
Based on my personal testing there is no additional work to be done in the realm of updating to async/await. All tests pass and the IRC server and matrix bridge function as expected. @jplatte kindly announced in the This Week In Matrix blog that we will be conducting public testing of the async_await branch on github. Barring any issues the async/await code should be merged into the master branch in the next few weeks.
TechRadar have published a fairly thorough Element secure messenger review. This covers the Web, Android and iOS clients, and appears to use the matrix.org server for signup.
The review is very positive, awarding 4.5/5, and concludes:
The Element messenger platform scores highly for its approach to security and its commitment to decentralization, and it's definitely going to be of interest to businesses wanting control over their own chats โ as well as plenty of individual users as well.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
This week we put out two point releases to fix critical bugs. At the very least ensure that you have upgraded to 1.19.2 which is a security release but you may as well go the whole hog and upgrade to the hot off the press 1.19.3. We will release 1.20.0 early next week.
Aside from that we continue to work on performance analysis and the sharding of the event persister continues - itโs proving to be a really tough job but we are getting there. Weโve also been making progress on the room knocking implementation
This week we say goodbye to Oliver (reivilibre) as a leaving present he fixed a long standing bug that prevented servers catching up after a federating outage and as we speak is furiously trying to finish a service to test TURN configuration. Thanks Oliver!
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go
Welcome back! This week we made some groundbreaking progress - this is not a joke, we found a bug in Synapse while breaking multiple Matrix rooms, causing the Matrix team a lot of unnecessary stress. Let me explain:
The good news is that Conduit is starting to federate now. This means that you should be able to join all public rooms of the Matrix network and exchange messages. Note that Conduit does not do all the checks it should be doing yet making it stop sending messages from time to time as well as that advanced features like loading the history, syncing temporary data like read receipts or accepting invites are not implemented yet.
The bad news is that, while Synapse is happy to accept Conduit's messages when it is already part of the room, joining into one of the rooms Conduit servers are part of didn't work because of an event validation bug. The Matrix team did an excellent job at fixing this bug and releasing a Synapse patch the same day, but the damage has been done making a few rooms inaccessible to old Synapse servers.
Thanks to everyone who supports me on "Liberapay" (https://liberapay.com/timokoesters) or Bitcoin!
I ported Synapse + its dependencies to NetBSD and it is now in pkgsrc as chat/matrix-synapse. This means that Synapse is now easily installable on any operating system that supports pkgsrc (which is many, e.g. NetBSD, Linux, macOS, Solaris, AIX, Haiku, โฆ). And while at it, I also ported mautrix-hangouts + dependencies (in pkgsrc as chat/mautrix-hangouts).
Hey bridge developer enthusiasts! Myself and ChristianP (bridge crew of matrix.org) have been working hard on freshening up the matrix-appservice-bridge library. It's no secret that it was using "classic" Javascript rather than ES6 syntax, was poorly documented in places and there were no types at all to make use of.
But that's all changed now, the 2.0.0-rc1 release brings fresh types, new convenience methods, a total swichover from Bluebird Promises to native Promises and much much more. If you are looking to quickly get set up writing bridges, it's never been easier.
Check out the slack-starter project to get your first taste of bridge excellence and see what you think.
As this is an RC, feedback is valuable and logging bugs helps us all. Please do so if you encounter any issues ๐
Hi encryption fans. This week the bridge team has been churning away at adding support for encrypted rooms in the matrix-appservice-bridge library, to enable support in matrix-appservice-irc, matrix-appservice-slack, and anyone else using the library. This is probably the most exciting feature the library has seen in a long time!
The feature will be a simple config toggle, so existing bridges will have to do very little work to support it.
The first phase plans to use Pantalaimon with bridge users syncing like real users, but eventually the hope is that this will be all built into the library. Come check out this issue to keep track of progress.
Element Android 1.0.7 has been released (still stuck in Google pipes writing those lines), and will be quickly followed by 1.0.8 to fix a problem with cross signing verification.
We are working to implement search of messages in a room and we keep on fighting bugs and improving performance of the application.
We are also iterating on the home screen, to improve user experience, and try to satisfy users with a few rooms as well as users with several hundreds of rooms.
Released 0.037 on Monday with some polish for E2EE:
remove outdated devices when querying
update room summary with decrypted events, also when retrying decryption
various crash protections
keep megolm session with earliest index when receiving a room key we already know about
add new icon
fix issue where olm would be loaded twice when login failed
show decryption errors in timeline
show encryption enabled tile in timeline
The rest of the week was implementing session backup, which I hoped would be ready to release today but it needs some more polish unfortunately. In the meantime, here's a GIF teaser:
Hemppa gained a small but big feature for making fediverse better: per-room Mastodon tooting support. This means that you can set up a Mastodon account in your room and anyone with moderator rights can toot to that account. This way you and your roommates can super easily send toots to the world directly from Matrix. Mastodon (unlike the big alternative) supports RSS feeds out of the box so it's very easy to subscribe to users and hashtags with stock RSS bot. Toot #hemppathebot if you like it! Another new contribution worth mentioning is welcome which can send welcome messages to users registering on the server or joining a room. https://github.com/vranki/hemppa
Another week, another Ruby SDK release. This time adding a little fix to avoid duplicate state events being passed twice to the application if identical ones arrive in both state and timeline, also adds a global state event handler and moves room event handling to not depend on room instances themselves.
Asked how much the SDK is used in production, Ananace said:
a little, we've hooked Matrix into our server orchestration / configuration management system TheForeman - which is a Rails application. Also got a colleague who's doing a bot to linkify internal ticket IDs.
Thanks to @q-b and @jtescher we have implemented more of the federation endpoints (bringing us up to about 85% completion) and every (one) endpoint of the Push Gateway API!
We fixed a bug where federation endpoints expected access tokens instead of server signatures for authentication. In the future, we may add an API that requires signatures on requests.
Finally, we've also continued progress on the non-exhaustive type updates for backwards compatibility; we expect to finish this within the coming weeks.
v0.2.0 of matrix-reminder-bot has been released! This releases includes lots of bugfixes, updates and polishing. Find the list below:
Features
Better support for command prefixes other than the default !.
Just writing !silence now silences the currently active alarm.
The bot will now print the correct syntax of a command if the user fails to follow it.
The bot will reply to events if it cannot decrypt them, as well as offer helpful tips that both the user and bot operator can try to fix things.
Bugfixes
Timezones. They should finally work correctly! ...no 100% guarantees though.
Alarms were a bit broken. They're fixed now.
Fix commands with formatting and newlines not being picked up by the bot.
Fix non-latin characters preventing reminders from being deleted.
Internal changes
Add a dev-optimised Dockerfile for quicker iteration during development.
Better wording revolving alarms. They're just reminders that alarm repeatedly when they go off.
Log why the bot is unable to start.
Don't print "Unknown help topic" in case the user is trying to ask another bot for help.
The config dict is now a singleton.
Type hints everywhere!
Additionally, the minimum Python version is now 3.6. matrix-reminder-bot is made with nio-template.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Normally the ping shows how long other servers take to receive the sending-server's message. Some servers receive it quickly, some take hours. All this flows into the statistic of the sending server. I wrote a small program that's the other way around. Instead of measuring the sending-server's ping, it associates each measurement with an echo-bot (instead of the sending server), so this plot shows which maubots are the fastest to respond (it's probably not accurate at all, because most of the pings were started on conduit servers this week, but it's interesting nevertheless):
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Wow the summer has flown by, it feels like just yesterday I was learning how to rebase and what exactly it is Ruma does. I exaggerate slightly, but it is a big library with lots of public API surface. I have learned more in the last few months than in two years of school. I have been able to observe and participate in a project with a community growing around it, been a part of discussions about design and best practices, given and received numerous code reviews as well as learned the process of addressing the feedback, and working from a specification. In short, this has been an amazing opportunity to gain experience in all the things that are hard to obtain in a classroom.
My project goal was to improve the existing macros in ruma-events-macros and ruma-api-macros. It became clear early on that this would include some major API changes and that improving the macros as they were was pointless without also moving to a new public API. While improving the durability and readability of the macro code I also rewrote entire sections to accommodate the new design.
A quick overview of the Matrix protocol for reference: a client sends content that is interpreted by the server as events. The server distributes those events out to other clients and other servers (the server case is known as federation). Ruma groups these events by kind Message, State, Ephemeral, ToDevice and Basic which are represented as generic structs (StateEvent<C>). Each event kind needs to be able to hold many different content types, for state events there is room creation, room name, and membership events to name a
few. Using the macros, enums are generated to represent all state event possibilities, so a variant for membership, room name, etc. These types exist to support the core API request and response types for each endpoint that is defined by the Matrix specification.
One of my personal goals was to become more familiar with git. With the help of my mentor I now feel more confident using this tool that is so essential to developers. I became fairly adept at merging, rebasing, and navigating all the headaches that come with that. I learned plenty of new commands. A few highlights: cherry-pick and specific uses of reset to avoid copy-pasting fixes and adding more commits. I used the reset command to craft good commits, splitting work into appropriate chunks. I am glad that I had the opportunity to hone my git skills. I feel like I have accomplished my goal and then some!
I am proud of the work that I have done: Being part of moving ruma-events much closer to the 0.22 release and creating macros to generate types specific to the Matrix specification. Working with the community that has grown around Ruma has been rewarding and I plan on sticking around.
Synapse 1.19.2 is a security patch. All federating instances should upgrade immediately.
Today we are releasing Synapse 1.19.2, which is a security patch release containing a fix to encountering invalid events over federation. We are also putting out a fourth release candidate for the upcoming Synapse 1.20.0 release with the same fix.
The bug prevents affected Synapse instances from joining rooms with invalid events. Server administrators running federating instances are strongly encouraged to update as soon as possible.
Those on Synapse 1.19.1 or earlier should upgrade to Synapse 1.19.2, while those who are running a release candidate of Synapse 1.20.0 should upgrade to 1.20.0rc4.
Get the new releases from any of the usual sources mentioned at https://github.com/matrix-org/synapse/blob/master/INSTALL.md. 1.19.2 is on github here, and 1.20.0rc4 is here.
The changelog for 1.19.2 is as follows:
Synapse 1.19.2 (2020-09-16)
Due to the issue below server admins are encouraged to upgrade as soon as possible.
Bugfixes
Fix joining rooms over federation that include malformed events. (#8324)
My name is Arnav Tiwari and I am a prefinal year undergraduate student from IIT Kharagpur and I wanted to share my amazing journey with Matrix. I am a budding open-source enthusiast and this was my first experience with Google Summer of Code. For the past few months, I had been working on a project to develop an HTML embeddable chat client under the GSoC program for Matrix. Matrix provides a highly versatile SDK for making custom clients that can be leveraged for a variety of applications, one of which is using Matrix to power an embeddable chat client. A chat client itself can have numerous forms, whether it being a live chat to a simple comments section. This project was intended to provide an easy-to-use and yet highly customizable client that can be deployed on a website with minimal effort.
My goal for the project was to have a useable project by the end of the coding period, however, as it turned out, the project was going to be tested in the real world far sooner than that. The need for an embeddable client and the feasibility of the project to fulfill this role was demonstrated during the second month of the coding period itself. The client was deployed on the website of CommCon 2020, a virtual conference on communication technologies (an apt place to be tested, coincidentally). On the days leading up to the conference and during the conference itself, I helped the organizers to set up, integrate, and troubleshoot the client when required. While the process went mostly without any hiccups, there was one small incident when the client broke during production. Since the project was still pretty early in development, I didnโt expect it to be bug-free and had anticipated the possibility of this happening. I was keeping an eye on things, which proved to be a prudent decision as I was able to fix this problem quickly and with minimal downtime. The rest of the conference went smoothly and the client performed quite well even when the number of users was quite large (A testament to Matrixโs scalability). Getting to experience this was a pleasant surprise since I never expected to have real users so soon, much less so many at once. Seeing the client being used out in the wild was a very fulfilling thing to witness. I also gained some very valuable feedback, courtesy of Dan, CommConโs master of ceremonies.
Over the next month or so, I kept on steadily adding features and building up the client. The next big break for the project came in the form of another conference. KDE Akademy 2020. This was a big surprise as I genuinely didnโt expect to see another large conference using the project so soon. The conference was scheduled to be held after a week or so after the end of the coding period. This time, however, the integration was almost completely handled by the conference organizers themselves since it had to be integrated with their version of BigBlueButton, a web conferencing system. As the conference drew nearer, things seemed to be working out well and there was no sign of trouble. When the day of the conference finally came, however, many things seem to break simultaneously due to an apparent incompatibility with BBB. In the end, despite the numerous attempts by the conference organizers and myself to remedy the issues, they had to roll back to an older version of the chat since the risk would be too great. The organizers were understandably disappointed since they had spent a while working on the integration and had seen the great potential of using this client in place of their old chat system.. Even though It was a sad conclusion to the journey, there were still many lessons to be learned. Most importantly, even though the client might work well in standalone circumstances, ease of integration might have some room for improvement. Open-source development never truly stops and I donโt intend to give up on this project. I look forward to constantly improving it and seeing more people adopt it.
These past few months were a spectacular experience. Even before starting this journey, I knew I would learn a lot but this still managed to exceed all my expectations. I got to learn things I never would have thought I would be able to experience under GSoC. I met some awesome people along the way, Iโm extremely grateful to my mentors Ben Parsons and Travis Ralston for being the best mentors anyone could ever ask for. They were always approachable and friendly throughout the entire program and I never hesitated before asking for their help. Without their guidance, all of this wouldโve certainly not been possible. Lastly, I want to express my gratitude to Matrix for believing in me and giving me the opportunity to undertake this project. The Matrix community is full of talented people who will go the extra mile if you ask them for help. It has truly been a pleasure working with them and I hope to continue working with them in the future. Cheers and hope to see continue seeing you all!
Since we suddenly have multiple projects that've separately gone on to trial our new and shiny Matrix-on-location for multiple respective use-cases, from conferences to anonymous - encrypted - support chats, we've had to end our soft-launch period very quickly.
Oh dear.
For minutes, confusion reigned in the chat about just what this meant! Fortunately, clarity arrived:
To clarify; Ending the soft-launch period means that we've now gone on to a full launch, with public announcements and everything else that includes
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, the widgets spec and related MSCs are still being completed, however is formally removed from the focus list for this week. In its place we've put something in a very similar vein: MSC1960 (OpenID Connect exchange for widgets).
We had another long overdue Spec Core Team retro this week. On the whole it felt very productive, and we've hopefully helped identify some key issues and actions going forward to help internally organise the way the team spends time on MSCs (read: unblocking review progress from the team).
We're still working out our exact strategy asynchronously at the moment, so expect more details to follow soon.
This week we put out a new release candidate for Synapse 1.20.0, highlights include shadow ban support, more async/awaiting and unread message counts being added to sync responses to support upcoming notifications work.
Aside from that Erik has been continuing to get event persistence sharding ready for shipping. He already has a working version, but currently all workers will run at the speed for the slowest instance. Once that is fixed we will put it on matrix.org to see how it performs.
Patrick has finished up on asyncing all the things, and is now studying flame graphs to figure out where all the cycles on the main process are going.
Andrew has been taking a look at the knocking MSC, expecting concrete progress in the next few weeks.
Brendan has been working notifications support and is currently in VoIP land helping Dave on trying to level up our VoIP support.
Finally Oliver has dusted off his TURN tester from last Summer to get it working before he returns to university. The idea is that it will act a bit like the federation tester for but for TURN configuration.
Hi everyone, Devin R and I are working on state resolution, hopefully we have something exciting to show you next week. I also worked on the remaining key backup endpoints, so for example deleting backups works as expected now.
Thanks to everyone who supports me on Liberapay or Bitcoin!
Dendrite
Dendrite is a next-generation homeserver written in Go
Released 0.0.36 with end-to-end encryption enabled! ๐
Most of this week was spent on performance improvements for IE11, where asm.js runs quite slow. Decryption is run in parallel on 4 workers in that browser, giving a massive speed-up and added responsiveness (the UI would freeze for 7 seconds when opening a room prior to this). There is still some polishing to do on sharing room keys, which I will get to next week, but all in all, it should work well. Next week I also want to get started on supporting key backup, so you can also read your encrypted history from before you logged in.
Element for Nextcloud (formerly Riot Chat for Nextcloud) has released v0.6.5 and v0.6.6. In this update, Element Web was updated to v1.7.5 and support for adding a custom integration server in the config through the settings UI and support for Nextcloud 20 was added.
We have released Element Android 1.0.6. Now we are working on sync mode for F-Droid version of the application (i.e. without FCM). We are also improving the experience with 1-1 calls.
This week, we released 1.0.10 which provides PIN protection and the return of the incoming native call screen. Then, we started to update room creation flows with new icons and behaviors for creation buttons and a new screen for room creation.
Maunium sticker picker
This isn't strictly a client, but it's in the clients section.
Getting stuff into the spec or even implemented in Elements is very slow, so while we wait for proper native sticker packs and pickers (i.e. not in an iframe/webview), I decided to make a better sticker picker widget: https://github.com/maunium/stickerpicker
The goal is to be as fast and simple as possible. The picker and sticker packs are just some static files that can be served anywhere (no compilation required). It doesn't require self-hosting anything except said static files, and it has been confirmed to work on all three Elements.
There's a Telegram import script that's used to reupload stickers to Matrix and generate the static sticker pack files that the picker widget uses. It's technically also possible to make the sticker pack files yourself, but there's no easy way to do that yet.
Due to the simplicity of the picker there's no authentication, which also means everyone sees the same sticker packs. One way to get around that limitation is to have a unique URL for each user. I'll probably write a bot to manage copies of the picker for multiple users in the future.
Other future improvements include remembering frequently used stickers (surprisingly, all three Elements keep localStorage for widgets), importing Telegram animated stickers and searching for stickers by emoji/name
later more words appeared from Tulir:
Since the initial announcement, my sticker picker has received some more features:
Added a slider to configure the number of stickers shown per row
Frequently used stickers now show up at the top
Imported all the packs from Scalar (the default integration manager)
Just bumped the Matrix Ruby SDK to version 2.1.2, fixing another bug with server discovery as well as the fact that state events haven't been provided for the application using the SDK. Oops.
libQuotient 0.6.1 is out, with fixes for bugs found during the work on the next Quaternion release and also an optimisation in the way profiles of all the users are managed - noticeable when opening large rooms (like our lovely HQ). Quaternion 0.0.9.5 beta is coming real soon now (TM), with some big features being coded in and merged as I write this.
I created mnotify which is a CLI based matrix client for automation tasks in shell scripts. Currently it supports login (via password), logout, sending messages, simple syncing, "get"ting the synapse admin api, creating a room, inviting users to a room, and joining a room.
https://publiclist.anchel.nl has been updated. The publiclist contains several public homeservers who can be used freely. Also listed some communities and roomdirectories which can be found at homeservers, last but not least public useable bridges can be found. The homeserver list contains now also uptime percentage and ip info. There is also a json that can be used by client builders.
And in completely unrelated news, I also just tagged version 1.0.0 of an invite bot for Matrix, which helps in doing bulk invites to communities and community-associated rooms.
A single bulk-invite (MXID or 3PID) to a main room will be propagated to invites into the community (if wanted), as well as invites to all rooms linked to the community, once the invited user joins the main room.
Michael Collins talks about the Matrix, covering the new features, perthchat.org, and ChatOasis an upcoming FOSS Matrix hosting company.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
325.5
2
helderferreira.io
614.5
3
neko.dev
661.5
4
kif.rocks
718.5
5
matrix.vgorcum.com
1785.5
6
an-atom-in.space
2034
7
urech.ca
2226.5
8
ragon.xyz
2436
9
im.kabi.tk
2478
10
utzutzutz.net
2588.5
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
I'm Tyagraj Desigar and I worked on Opsdroid with Stuart Mumford and Drew Leonard. Opsdroid is a python framework for creating platform agnostic bots usable with multiple chat services including matrix.
The Plan
The project's focus was on the interaction layer (called a connector) between opsdroid and matrix. As it stood, the connector used the deprecated matrix-python-sdk and had no support for encryption. The aim was to change this by moving over to matrix-nio.
I had planned to work on some other features as well:
A database module for opsdroid using matrix state events
Support for unused matrix events
Homeserver lookup using .well-known API requests
Device verification process for the bot
Porting to nio and adding encryption support was the bulk of the project, and the minimum I wanted to accomplish by the end of the project. I wanted to have a connector that worked as it did already with support for encrypted rooms which needed minimal extra configuration.
The Process
The process began with This PR which gave us a head start with the migration to matrix-nio. Through helping to review the PR I got to understand how matrix-nio and opsdroid work.
The implementation of encryption support was a little tricky in that it was tough to figure out the process required to do that in the context of the connector. One problem we faced was the installation of dependencies. Installing libolm, a C library which nio uses for encryption was a less than smooth process. This spawned a couple side projects that dealt with the CI, testing and installation of opsdroid. In the end we found a solution and had a working connector on our hands.
Next we shifted focus to the database module. It was based on this project. I rewrote it with nio and added a few features. The idea was straightforward but the implementation had many catches since we were working under some constraints for the interfacing with opsdroid. It went through several iterations before we settled on the final product.
The encryption and database took longer than I had initially expected which meant we didn't get to work on adding support for more events and homeserver lookups. I had a go at adding device verification steps while working on the encryption support but that turned out to be quite complicated and would introduce some breaking changes, besides cross-signing had just been introduced so we decided to drop that till nio implemented some way to leverage cross-signing.
Along the way some issues cropped up with the testing frameworks and CI that were hard to pinpoint and caused further delays but were solved eventually.
The Conclusion
I am extremely happy with what I've accomplished and hope that I have been able to achieve the standards set by the matrix community. It was a challenging and exhilarating experience. I have learned more in this project than I could've imagined and gained a ton of invaluable experience with software development thanks to my mentors who pushed me forward and guided me throughout. This journey was beyond amazing and I will be sure to contribute to matrix moving forward.
You can find complete details of everything I worked on during GSoC here.
I'm CH Chethan Reddy. I'm currently pursuing Bachelors in Electronics and Communication Engineering at the National Institute of Technology, Trichy in India. In my free time I like to swim, work on projects I find interesting, watch movies and do a lot of other "interesting" stuff. This summer I have interned in Matrix.org for around 3-4 months under GSoC and in this blog I'll be sharing my experience.
Why Matrix.org?
Being passionate about privacy, I wanted to select an organization that actively works on that, and clearly, Matrix.org topped my list. For anyone who doesn't know what Matrix is, in simple words, It is a decentralized communication protocol that supports many awesome features like messaging, end-to-end encryption, voip/video calls support, etc. It uses simple RESTful http APIs which keeps things very simple. The best part of the protocol is, it is decentralized, so if you have a custom domain name, you can setup a synapse server in a jiffy. Knowing all these who wouldn't choose Matrix?
My Work
I worked on Desktop Client called Nheko. It is a light-weight Desktop Client that uses Qt Framework in the frontend and also uses a mtxclient library that implements Matrix-client server API written in C++. The scope of my project involved implementing Device-Verification and Cross-Signing in Nheko.
For people who have not used Matrix before, it is important to note that every device of a particular user is considered as a separate unit rather than user itself. For end-to-end encrypted chat to be viewable from a device, that particular device should be trusted by the sender's device. Device-verification and Cross-Signing are methods used for verifying.
Coming to the work I have implemented, Device-verification is completely implemented in Nheko. As of now, SAS verification with to-device and room message verification is supported in Nheko. As far as Cross-Signing is concerned, verifying signatures of Cross-Signing keys and showing the verified status of devices have been implemented. The only remaining unfinished parts are SSSS and signing the keys after verification. After that Cross-signing should be feature-complete.
I wished to implement SSSS during my GSoC period, but I couldn't because I did not anticipate the additional things which came with verification while making my proposal. The additional works included : implementing the UI, changes to the userprofile dialog, working on the caching of verified users, and user keys. Moreover, due to the current pandemic, there were a few sudden changes in my academic schedule, which interrupted my work to some extent. While contributing at times I was stuck on some bugs like verifying the signatures, some random crashes in UserProfile and setting up relations in Room-Verification. However, with the help of my mentors, I was able to fix these bugs.
I'm really happy about the work I have done. I am hoping to further work on Nheko in the future, complete Cross-Signing and work on additional features for Nheko.
Community and Mentors
The best part of my experience this summer was the the Matrix community and the learning I had. I am really lucky to be part of the Matrix community which has many passionate people collectively working on really cool projects with clients, bridges, servers, bots and the spec. A special thanks to Uhoreg and Sorunome who have helped me in navigating through the spec.
Last but not the least, a big shout-out to both my mentors, Nico and red_sky, who were always there to help me with any issue in the project in spite of their personal commitments and dealt with my stupid questions with utmost patience and kindness. Without their active help and guidance this experience definitely would not have been so fun and great, they have clearly surpassed the expectations I had from mentors.
Hello! I am Nikos, an MSc student in Computer Security. My original goal for this GSoC was to implement the basics of end-to-end encryption for Go-NEB, which is a popular and easily extendable bot for the Matrix protocol. I chose this project because it ticked off a lot of the boxes that I look for in a project: The Matrix protocol is very interesting and the e2e specification is fascinating, Go-NEB is written in Go which I wanted to learn better, plus I believe personally that crypto should be easy for everyone to use and adopted more.
My initial plan was to first create bindings in gomatrix for libolm, the C library that implements most of the necessary algorithms for Olm/Megolm, the protocols used in Matrix' end-to-end encryption. Then, I was going to write some nice APIs around those bindings so that most of the work done is hidden from the client that uses the library, and finally I would call these APIs from Go-NEB when needed to set up the encrypted sessions between devices in a room and use them to encrypt/decrypt messages as necessary. What we found out soon after GSoC started was that another library called mautrix-go that was based on gomatrix already did most of these things. My task then became to change the uses of gomatrix in Go-NEB for mautrix-go and make some slight changes to the latter if there were some incompatibilities. A month later, that task was done.
With two months to go in GSoC, my mentors came up with a new task: to implement a service for Go-NEB that allows other client developers to test their e2ee implementations easily. That resulted in a service called "cryptotest" which allows other clients in a specified room to execute some Neb commands which would trigger functions like the forwarding of keys between clients, another feature that had to be implemented in mautrix. Something else that I wanted to add and grabbed my interest was SAS device verification, a multi-step process that involves users comparing a string of numbers or emojis out-of-band to configure they are indeed talking to each other and not to a MITM. These were my main tasks during the second month and I was happy to have achieved them and made a decent verification API for clients to use. They were soon merged into mautrix and Go-NEB.
For my third month I proposed to implement a new Matrix feature called cross-signing, which was similar to SAS verification but dealt with verifying other users instead of their devices who would in turn verify their own devices, creating a graph of trust between users and devices. This functionality was strongly coupled with SSSS, another new feature that allows clients to store their encrypted secrets (in this case the multiple necessary signing keys) on the server, as well as in-room verification which allows verification between users. This task was more challenging as it uses multiple algorithms (some of which not in libolm) for deriving the keys, using them to encrypt other keys which in turn sign other keys. It was also more satisfying when I finally managed to generate the keys, store and retrieve them and then use them to sign another user and when I was done I felt like my understanding of cryptography levelled up.
All in all, I'm very satisfied with the overall experience. The spec was very clear in most cases and when it wasn't the community was always helpful and responsive. They were also happy to discuss the spec with me and explain the more intricate details. For that reason I wasn't asking questions directly to my mentors (besides for defining my tasks) but to the overall community. I'm also very grateful to my mentors, @uhoreg and @Kegan for picking me for this project and helping me with the planning aspect and reviewing my PRs. I'd lastly like to thank the maintainer of mautrix, @tulir, for reviewing and merging a lot of my work into his library.
The Matrix service at Linkรถping University has now passed its soft launch state and headed straight into full and proper deployment territory. (Though we're still not being overly vocal about it, don't want to crash our deployment on day one after all.)
They're simply announcing it on TWIM, perhaps the most high-profile place in the world!
We're doing a little bit of advertising of it to computer clubs and the like though, to slowly build up the users.
But then:
And just to add another slight update: already we have at least one conference planning on trying out the new Matrix server as well, so the user count might just end up exploding soon anyway.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
In terms of Spec Core Team MSC focus for this week, we're sticking with the widget theme, but also adding something new. Travis is writing up a mahoosive widgets PR in addition to some widget MSCs. We're also adding on MSC2753 (peeking /sync), MSC2444 (peeking federation), which will both unlock some very exciting upcoming features (such as x-as-rooms)!
Erik has made an initial, naive implementation of a sharded Event Persister worker - the non-sharded version currently being one of the biggest bottlenecks on matrix.org right now. It still needs a bit of polish to be production ready, and Sytest will need some updating to support testing it, but otherwise progress is looking good!
Brendan is polishing his unread counts implementations, while Rich is writing up the push rules replacement tech in an MSC. Rich has also been optimizing the device code in Synapse to both simplify the code, as well as cut down on unnecessary device data being sent over federation.
Patrick has finished converting Synapse's database code to be fully async! This has the benefit of making profiling the matrix.org Synapse process much more sane, and we're already starting to find a few little places where Synapse is burning CPU unnecessarily! Now, Synapse won't be fully async'd until Twisted itself is, and we drop support for older Twisted versions that aren't, which is not soon. However the majority of the code that developers touch and that is executed being async has massive benefits either way ๐
Andrew has been working on having a confirmation pop up after users click the link in their email, to help prevent phishing scams. He's also trying to finally make Synapse spec-compliant in accepting identifier dictionaries during user-interactive authentication.
Olivier has been continuing to help Synapse recover missed messages better after it has been offline for an extended period of time, as well as various fixes to push.
Expect a fresh release with all these changes soon!
As someone who had my own server outage this week I'm looking forward to seeing this!
Dendrite / gomatrixserverlib
Dendrite is a next-generation homeserver written in Go
This week we have been continuing on our bug-hunt mission and have been working on refactoring some significant parts of the codebase to simplify the API surfaces and avoid race conditions.
Changes this week include:
The current state server is in the process of being deprecated, with this functionality being folded into the roomserver
Support for rate limiting on certain CS API endpoints has been added
Support for password changes has been added, allowing users to change their own password
/sync no longer incorrectly lists some rooms in the leave section, which caused problems with Element Web and other clients
Roomserver input events are now handled in FIFO order with per-room workers
Roomserver input, query and perform packages are now split up nicely
The federation sender and keyserver now delay some startup tasks to allow the HTTP listeners to start more quickly
The federation sender is now used for backfilling and getting missing events
Some more SQLite writer bugs have been fixed
Room information in the roomserver is now handled in a common structure
The public rooms API now correctly searches remote servers (thanks rohitmohan96)
If contributing to Dendrite sounds like something you would be interested in, please take a look at these issues and join us in #dendrite-dev:matrix.org! There's also #dendrite:matrix.org for general Dendrite chat and updates.
Introducing a new bridge to the family, tomsg-matrix!
A friend of mine preferred using his own chat protocol, but having a chat protocol is useless without having people to talk to. Since I prefer using Matrix I thought it would be a fun exercise to try and create a bridge using Rust.
We are currently working on an Android-App to use an old smartphone as SMS-Gateway for matrix-sms-bridge, because using gammu in production (>50 SMS/day) seems not to be very reliable with our USB-modem.
I create service for sending corporate information: some service (for example zabbix) can send message to this service (over http-api) and service will create room (if needed), invite user and send message to this room. Service will change status of message in his database to readed when user will read this message. Also service can send any changes statuses of this message by callback url. Editing old messages also supported: for example zabbix can send UID problem to this service and it is will edit old "problem message" to "problem resolved".
I'm currently in the process of rebuilding Ditto - partly to incorporate rn-matrix, and partly to support a WEB VERSION
The new repo can be found here (https://gitlab.com/ditto-chat/ditto) (there's a new repo mostly because getting react-native-web to play nicely with the rest of the app was a challenge, to say the least)
Good progress this week, I just got end-to-end encryption working. The code still needs some cleanup and polishing to be able to release it in the wild, but expect a release somewhere next week with E2EE enabled.
gomuks now supports importing and exporting message decryption keys. It even supports exporting the keys for a specific room, e.g. for sharing them with someone else. To help with sharing exported key files with someone else, gomuks also got support for uploading media.
Next I'll try to figure out how nikofil's cross-signing/SSSS PR for mautrix-go works and use that in gomuks.
NovaChat is a new Matrix-based desktop client that aggregates all your chat networks into one app. We now have integrated bridges for Whatsapp, Twitter, Telegram, Hangouts, Slack, Messenger, Android Messages (SMS), Skype, Discord, Instagram and IRC.
Sept 4 Updates:
Continuous UI improvements. Everything is much denser now. See screenshot below.
New bridge alert: tulir just completed an Android Messages SMS bridge (which is a total hack but it works!) for NovaChat.
Added Discord Bridge. Thanks for building this Sorunome!
In the works...
Still working on our iMessage bridge, using jailbroken iPhone 4s
Marking messages as 'unread'
Sign up for the NovaChat beta. Or send me a DM @eric:nova.chat!
Nheko
Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE (with the notable exception being device verification for now) and intends to be full featured and nice to look at
Lurkki added an emoji completer to the message area! This was a super requested feature and it is pretty awesome, that someone picked this up and implemented it! This means you can finally write :fire: and have it expand to ๐ฅ! Seems like this will also lead to a few more improvements to our other completers down the line (user, room, etc completions), so thank you very much Lurkki!
I've been mostly reviewing PRs this week (and trying to get some other work in line, like MSCs, synapse stuff, etc). With this hopefully Chethans work on cross-signing will be merged soon, which will give us most of the building blocks to have cross-signing in Nheko (his work is basically only missing Secure Secret Storage and Sharing). We hope to fix the few remaining build issues in the next week or so.
On the side I have been cleaning some UI stuff to simplify our room settings dialog and allow advanced users to hide/show certain event types in the timeline. This will probably take a bit more time to be finished though.
IMPORTANT: Due to legal reasons (USA cryptography export), end-to-end encryption has been disabled in the google play and ios builds for now. They will be re-enabled once we got all that figured out.
The F-Droid builds still have e2ee. As the play store and fdroid builds use the same build signature, you can simply install the fdroid version over the other one. More information on that here: https://fluffychat.im/en/fdroid.html
Features
Pin and unpin chats
Implement event aggregations
Implement message edits
Render reactions
Add / Remove reactions by tapping on existing reactions
Fixes:
Don't re-render the room list nearly as often, increasing performance
Various fixes for sending messages on bad networks
I removed SchildiChat from the Google Play for now, due to the complications of providing encryption-capable software from US servers. Huge thanks to the FluffyChat team for providing insight into their investigations concerning the US export regulations!
On the positive side, SchildiChat is now finally available on F-Droid from the official repos.
Furthermore, I have set up my own F-Droid repository, where you can get updates that are compatible with the Google Play Store variant and which support push notifications.
On the implementation side, I can report the following:
Element 1.0.5 has been merged
The "low priority" category now remembers its expand/collapse-state across restarts
New setting to hide member state changes and reactions from the chat overview
The category unread counter badge now also displays the number of chats with new messages in case none of these has notifications enabled
This week, we should have fixed the background crash that was happening in TestFlight 1.0.7. TestFlight 1.0.9 has been submitted yesterday.
In parallel, we started to modernise the AppDelegate class, the entry point of the app. This is the very first step of several improvements we want to do (this is not only to rewrite it in Swift).
We have just merged a few big PRs to develop: add emails and phone numbers to Matrix account, rework of upload attachment management, and other stuff related to 1-1 calls. It will live on develop for the week-end and we will prepare the release v1.0.6 at the beginning of next week.
Built on the matrix-js-sdk, this NPM package will help any React Native developer drop matrix into their app.
Been working on this since the beginning of the summer - this package is split into two, one for the data and one with default UI components for those who want to drop chat into their app quickly (not required).
Would love feedback on docs, installation, and usage, so try it out and drop me a line!
Over the past couple of weeks, jplatte has been hard at work making sure we can introduce new fields to request and response types as non-breaking updates. Thanks to @nicholaswyoung, consumers can opt-in to breaking changes to ensure compliance with the spec on each update.
Besides that, we implemented another endpoint for the Federation API, bringing us up to 20/31 endpoints implemented.
matrix-docker-ansible-deploy can now help you set up matrix-registration - an application that lets you keep your Matrix server's registration private, but still allow certain users (those having a unique registration link) to register by themselves.
The interest was high. Some didn't know, that bridges exist in Matrix.
!!! Well thank you Oleg for letting them know.
Dept of Ping ๐
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server. Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
fairydust.space
376
2
neko.dev
444.5
3
tchncs.de
565.5
4
autodie.be
958
5
matrix.vgorcum.com
1124
6
maescool.be
1435
7
lermer.nl
1989
8
utzutzutz.net
2207
9
bau-ha.us
2965
10
rollyourown.xyz
3013
That's all I know ๐
See you next week, and be sure to stop by #twim:matrix.org with your updates!
