Discussion
Following the mastodon 5.0 #uxfail saga on https://community.joinmastodon.org has been interesting. Part of community building is accepting feedback on changes and I DO NOT envy the mastodon team that job. There is always a certain amount of "entitled users, screaming about any change at all". And a lot of feedback is of the form of "I think it should look like this instead...", which is frankly, not very helpful to designers.
1/
But what all this has highlighted is the backward-ness of how they made these changes. 1 week of workshops, followed by testing on 1M users, followed by soliciting feedback on a forum is not going to produce actionable data.
User testing's bread and butter is observing user behavior 1 on 1. Not soliciting feedback, observing what users do. This results in objective data that can be used to make informed design decisions. This testing needs to be done with a representative sample of users.
3/
@intrepidhero FWIW, it was actually more like this: Sorting through existing GH issues, reading 3rd party academic reports about Mastodon user challenges, 1 week of workshops + surveys, reviewing feedback after the August blog announcement, prototyping + user testing (+ reviewing video recordings), then testing in beta.
Before this testing can happen the application's interaction space needs to be rigorously mapped. What's can the user do? is the first question to answer. Then testing answers, How do they (in the current or proposed design) accomplish those tasks?
Only then can iterative design and testing attempt to streamline those processes.
4/
Subjective feedback is useful during this process but data is the primary driver. A user may have good suggestions, but with the firehose that the mastodon team turned on, god help them find the diamonds among the chaff. And without a rigorous approach they have no way to evaluate suggestions or changes.
5/
@intrepidhero The first challenge, as you mentioned, is that we're extremely under-resourced. One designer for over 8 thousand servers and 800K MAUs. No product manager, no research team. We still have release timelines on par with much bigger orgs, which impacts our ability to receive grants and donations that keep Mastodon running.
2/5
@intrepidhero Secondly, defining "representative sample" gets trickier in the decentralized social space. What appears to be an objective truth in one corner of the fediverse may have a complete antithesis for Mastodon users elsewhere. If we had 10x the resources, solving this might look like having 6+ months to partner with server admins to source a diverse pool of consenting participants, and reviewing recordings to develop an understanding of common interactions, flows, and personas.
3/5
@intrepidhero Third challenge: As you said, telemetry would be "better than nothing" – I believe it would at least alert us to instances where the software is being used against our current assumptions, which could help us to narrow/prioritize research questions. But Mastodon's software collects as little user data as possible, which makes it nearly impossible to draw conclusive and statistically-significant insights.
4/5
@intrepidhero I realize these problems aren't unique to Mastodon, but are common challenges for designers of open-source software. How do we responsibly iterate on under-resourced, high-visibility projects without burning out? What are some low-telemetry ways we can find meaningful signal? I'd like to continue that conversation with others who are interested.
5/5
