”Bluesky controls the AT Protocol right now, and Graber is keenly aware that isn’t tenable if it’s going to be something lots of other companies build on over time. She says the plan is to eventually hand over control to a web standards body like the Internet Engineering Task Force. And as to whether her team plans to interoperate somehow with ActivityPub, she says it’s not on the roadmap but the company isn’t opposed to the idea.
“I think it’s a time of experimentation,” she says. Companies like Meta “are making moves that were unthinkable a few years ago” by embracing decentralization, and even though ActivityPub has captured the zeitgeist now, Bluesky thinks there’s still room for another take on what is increasingly looking like the next phase of social networking.”
Regarding #Bluesky opening signups, one person on HN comments:
> The account portability is probably the biggest problem with the #fediverse right now.
And while that may be true from an end-user perspective, imho it is not the biggest problem. The fact that for a new dev building an #ActivityPub app is like a Viking trying to discover America is problematic. While #ATProto has the Holland-America line where you can check in for the ride.
People who say things like "the Bluesky team doesn't seem to be in a hurry to open federation" or "why aren't they adding more features faster, what are they waiting for?" have no idea how much super hard and complex work is being done behind the scenes by a handful of very smart people… https://blue.mackuba.eu/skythread/?author=jaz.bsky.social&post=3kkjrmk2bzs2z
The paper figure is a lot cuter, but by linearizing it and presenting it as two parallel tracks they have obscured the most salient feature of the network: the big relay in the middle. Beyond "centralization bad," that pins down most of the undesirable and dangerous features of the protocol, and makes it seem like theres a lot more choice than there is.
Since the design purposefully hides the architecture: you dont know where your feed generators are drawing from, or those used by your friends. So you cant know what the effect of choosing a different relay would be, aka the main relay is always indispensable. Importantly the relays subscribe to you, you dont push to the relay, and since you arent really supposed to operate your own data store, you can be dropped from the network without knowing - the relay serves as an unaccountable point of moderation.
Its this part for me that tells the whole story: #atproto was designed for a new kind of advertising market, and when their VC money and puttering domain registration revenue streams dry up, control over the main firehose relay is a big gaping profit vector waiting to be capitalized on.
(One correction, about "There will probably only ever be one Bluesky app view" - hopefully not! It will be challenging to run one, but hopefully there will be at least a few competing ones eventually, otherwise it won't really be fully decentralized)
Welcome to a new short series on Bluesky and how the network works. Bluesky recently releasedmore information on their plans for third party moderation services. While writing about their plans, I realised that to properly explain how it works, I first needed to explain how the network is designed to function.
Most people understand the fediverse in terms of separate instances. Every instance can be a social network in itself, and by connecting with other instances form a larger network, the fediverse. This makes it easier to understand where content moderation happens: every instances has their own content moderation, own moderators and their own rules.
The Bluesky network and the AT Protocol function differently. There are different types of servers; servers for data storage, servers for data aggregation, etc. As such, content moderation happens in different places on the network. To properly explain how it works, the benefits and tradeoffs, as well as the unknowns, I am publishing a short series on Bluesky, how the network functions, and how and where content moderation happens.
In this first episode: the parts that make up the network and allow it to work.
The basic components
The Bluesky network consists of the following parts:
A Personal Data Server (PDS) that hosts all account data. It contains information about your accounts, and is where all your personal data is stored.
A Relay looks for all the PDS’s in the network, takes in all their data, and merges it together to outputs one big stream that is used by other parts of the network. The Relay puts out the data in a machine-readable format, which is often called a firehose.
AppView takes the data from the Relay, and processes it so that it is more meaningful for apps. Examples of the processing that AppView does: counting the amount of likes that a post gets, collecting all replies to a post and organising them into a thread. It also generates your “following” feed, by creating a reverse-chronologically ordered feed of posts made by the accounts that you follow.
An app, whether that is the official Bluesky mobile app or a third party website like deck.blue. The app takes the data from AppView and presents it in a nice format for people read on their preferred device.
With these four components we can imagine a basic social network:
If you open the official Bluesky app on your phone and look at the “following feed”, the data flows as follows:
All PDS’s => Relay => AppView => App.
If you then create a post and hit send, data goes from your app directly back to the PDS where your account is hosted.
Custom feeds and moderation
There are four more components to the Bluesky network: feed generators, labellers, the moderation service, and the Identity Directory.
A feed generator creates the custom feeds, using some form of algorithm. These custom feeds can be anything from a feed with the posts with the most likes in the last 24 hours, a feed of posts that contain specific terms, or anything else.
A feed generator takes data from a Relay, performs the calculations to take the raw data into a custom feed, and sends it to the AppView. The AppView then performs some final steps and sends it to your app so you can see the custom feed.
Labellers. Labelling services perform moderation activities by applying labels to a post. People can determine how they want to handle labelled content. An example of a label can be ‘Sexually Suggestive’, and people can determine if they want their app to either show, hide or warn about posts that contain the label.
A Labelling service takes data from the AppView, processes it, and then sends it back to the AppView
The moderation system, called Ozone, that allows moderators to take moderation action, such as taking down posts or accounts. This tool has the least amount of information on it, and it is not visible Federation Architecture documentation. The update this week by the Bluesky organisation shows that the system is called Ozone, and that they are in the process of making it open-source and available for others to use.
The tool at least allows moderators to alter data in the PDS, as that is where account data lives.
Every account on the Bluesky network has a unique identifier, called a DID. A DID is a unique string of random numbers and letters, and cannot change. Every account also has a handle, which is your username. New accounts start with youraccountname.bsky.social as a handle. The network also allows you to change your handle to a domain name that you own, which allows for easy verification. The information about which DID corresponds to which handle is stored in the DID PLC Directory.
Now we have all the components that together make up the Bluesky network. In the next part, I’ll take a look at decentralisation and federation, explaining for every part how it will play a role in decentralisation and federation.
Thanks to Kuba Suder for feedback on a first draft.
"ActivityPub and ATProto break #siloing in different ways. #ActivityPub is built around URLs and can "socialise" more or less anything on the Web, which is great, but they don't touch the underlying substrate—either you run your own server or you…are at the mercy of an admin. #ATProto, on its side, provides a good initial foundation for an extensible #PDS designed around user agency and credible exit.
…you can be guaranteed to be able to take your content elsewhere." https://berjon.com/ap-at/
So OK I keep trying to like #atproto / #bsky, lexicons specifically but they make it so hard. like... let's see, OK we're going to reinvent RDF and JSON-Schema such that
there is some concept of a "namespace authority" rooted in DNS ownership of a domain name that is supposed to maintain a set of lexicons and do things like make sure there are no duplicates (eg. lexicon IDs are case sensitive but you are not allowed to have IDs that differ only in case, which a ns authority has to ensure)
but there is no means of using the domain to retrieve a schema, so eg. it is explicitly not spec'd to require com.example.myLexicon be at myLexicon.example.com - so the whole goofy Javabrained reverse-domain name thing as the ID is pointless
Accordingly, lexicons have to be sourced ad-hoc and by convention, and the convention is bonkers!!!! According to their reference implementations, the way you are supposed to use lexicons is to autogenerate their json schema definitions and version them locally everywhere they are used. I can't express how baffling this is to me - like the atproto monorepo has a top level lexicons directory, and then their Makefile then goes through several of the subpackages calling each of their (separate) code generation scripts, resulting in local, vendored copies of each of the lexicons - and by copies of the lexicons i mean both generated code AND typescript interfaces AND an entire copy of the whole schema.
This is presumably because there are REFERENCES between lexicons EVEN WITHOUT those lexicons having a meaningful notion of location or dereferenceability - how would you know what 'app.bsky.feed.defs' is referring to unless you packaged and built all lexicons together? the method for resolving references literally requires them to all be locally present at runtime as far as i can tell, idk the lexicon code is unreadable to me.
There is no means of versioning, so "Once a schema is published, it can never change its constraints" and the only means of changing is to change its freaking name, so com.example.myLexicon.FinalFinalv2 is explicitly encouraged. The reference implementations repeatedly violate this, though, so whatever.
Again, there is no actual mechanism of "publishing" a schema, though, so it's completely unclear what that's supposed to mean - how do people know the schema has changed names since they just have it hardcoded in their clients?
and the entire spec documentation is just a bunch of harebrained ideas that amount to, as far as i can tell, just literally being JSON-Schema just different? And they compare it to RDF but there's literally nothing to compare there except for the absence of an explanation as to why they didn't just use JSON-LD. I literally cannot tell how this is supposed to work as an interoperability/extensibility layer if there is no means of resolving terms or lexicons and all definitions have to be known in advance.
Yes, it’s just Day 3 of 2024, and I’m already giving you work to do.
Today, we cover three neat, new-ish, services you can self-host to either use directly, or hack on to customize.
The decentralization movement is picking up speed, non-tech-y humans are also starting to switch to “lurker” mode vs. provide content to Big Tech for free, and organizations like Google and Amazon are starting to clean house by killing off apps and services that do not generate ad revenue.
This makes 2024 an ideal year to exercise some self-hosting muscles!
(IT IS SO NICE TO HAVE PROPER CODE BLOCK SUPPORT AGAIN)
The post discusses the importance of self-hosting in 2024, focusing on three services: Feedi, MicroBin, and skeetgen. The author emphasizes the growing trend of decentralization and the shift of users from content providers to lurkers. The post also highlights the actions of big tech companies like Google and Amazon, who are discontinuing apps and services that do not generate ad revenue.
The first service, Feedi, is a self-hosted feed reader developed by Facundo Olano. The author provides a detailed overview of Olano’s journey, from career burnout to rediscovering the joy of software development through the creation of Feedi. The post also discusses the broader themes of the software industry, the need for an open and independent web, and the quest to regain control over information in the digital age. The author provides a technical overview of Feedi, including its features, setup, and potential for customization.
The second and third services discussed are MicroBin and skeetgen. MicroBin is a self-hosted pastebin that offers features like server-side and client-side encryption, file uploads, raw text serving, QR code support, URL shortening and redirection, and private and public uploads. Skeetgen, on the other hand, allows users to generate a publicly viewable archive of their Bluesky posts on their own kit, providing an opportunity to learn about web apps, a new Bluesky/ATProto JS library, and SolidJS.
Before we get to the tech, I want to take a moment to give a quick overview of why @facundoolano decided to take on the project/journey, as it may provide some inspiration for folks at the start of this new year.
In “Reclaiming the Web with a Personal Reader,” Facundo Olano delves into his personal journey, marked by a phase of career burnout, which led him to rediscover the joy in software development. This reconnection was fueled in a project to create a personal feed reader (the aforementioned feedi). The blog post isn’t just about building some new side project, though. Olano hits notes on broader themes such as the nuances of the software industry, the burgeoning need for an open and independent web, and an introspective quest to regain control over the influx of information in today’s digital age.
The story begins with an awesome decision to quit Twitter and migrate to Mastodon. This shift was pivotal, as it dropped the author hip-deep into the IndieWeb movement and the concept of social readers, which influenced the vision for feedi. The intent was to craft a feed reader that not only mimicked the home feeds of Twitter and Mastodon but also maintained a consistent aesthetic and functionality, irrespective of the data source. Python was the chosen language for this endeavor, (despite the baggage of environmental and dependency hades challenges).
In designing feedi, the Olano opted for a monolithic web application structure. This structure incorporated server-side rendering libraries like htmx and hyperscript, supported by a SQLite database. Interestingly, the author chose to initially forego writing tests, prioritizing rapid development and experimentation instead. This approach reflects a blend of pragmatic and creative software development philosophies.
The culmination of this journey is feedi itself, which, for several months, has served as Facundo’s personal “front page of the internet.” This creation is more than just a technical achievement; it represents a curated, personalized approach to information consumption, challenging the status quo of how we interact with the web. Through feedi, the author not only rekindled their passion for software development but also reshaped their own digital experience.
Now, there is no requirement that every side project must have such an origin/back-story. And, it’s 100% cool to have tiny ones, vs. some monolith like feedi (it’s not that yuge). But, being able to peek behind the curtain to see the “why” and not just the “how” was refreshing.
Back to the tech drop.
You can see an Arc split-view of an instance of feedi running on my MBPro in an Orbstack (docker) container up in the section header.
Since I’m a big fan of temporal feeds, the idea to forego the “email client” look/feel that virtually every other feed reader has was super intriguing (though, I pretty much use Inoreader this way already, often hiding the sidebar and just scrolling down the temporal feed). The other core features of feedi include:
Easy local and self-hosted environment setup.
Mobile optimized UI.
Mastodon home, hashtag, and notification streams support.
Custom parsers for Reddit, Github and Goodreads.
Hackable RSS parsers and ad hoc scrapers.
Smart feed sorting options (highlight infrequent sources, auto mark as read).
Local article reading and preview using Mozilla’s reader mode.
Send to Kindle device support.
I did the:
$ git clone https://github.com/facundoolano/feedi.git$ cd feedi$ make docker
dance and launched a local instance in the background:
I’m using an alternate port since I already have bits running on 5000. Just hit localhost:#### and you’re good-to-go. You can also just run it without a container.
You can authenticate to as many Mastodon instances as you like, and you have full Mastodon features like boosting and fav’ing. Within each entry (Mastodon or regular RSS) you can see the JSON for the feedi record and for the source RSS. And, there are many keyboard shortcuts for power users.
You can even import any existing OPML (or CSV) feed files.
I’ve begun to set up “buckets” (folders) for GitUgh, Reddit, Mastodon, and “Newsletters” to get a feel for the app. While it would mean “work” (and, work in Python, no less), This could be a solid replacement for something like Inoreader or Feedly. The Python foundation would also make it possible to add in some advanced NLP (trying hard to not just slap an “AI” sticker on everything) features, and it would also be pretty simple to hack-in some of the advanced notification features I rely on in Inoreader.
There is no authentication by default, but the repo has full instructions for enabling that if you want to expose this to the big, bad internets (I suggest using a Caddy reverse proxy).
You can set it up via Docker or just cargo install microbin plus some environment variable tweaking. I’d suggest not using Bash/curl Docker method as that’s opaque (I know you aren’t going to read the main and downstream scripts before running it).
It has what one might expect from a modern, self-hosted pastebin:
Entirely self-contained, single-file executable
Server-side and client-side encryption
File uploads
Raw text serving
QR code support
URL shortening and redirection
Private and public, editable and uneditable, automatically and never expiring uploads
I’d highly suggest just using this as a personal or team pastebin via Tailscale vs. slap it on the internet.
The section header is a snapshot of it running locally in an OrbStack (docker) container.
Short and sweet, skeetgen lets us generate a publicly viewable archive of our Bluesky posts on your own kit, vs. rely on the finally available public Bluesky post viewing feature.
The challenge is that the baseline features of activitypub are essentially impossible to build an effective client off of.
It's not just they are defining their own interfaces, it is that they have to define their own interfaces if they want to be even reasonably effective at their chosen tasks at scale because of the limitations on how collections can be dereferenced
There are ways around this with FEPs and such, but it's not in the protocol
Checking in on whether #bluesky / #atproto has become any more like a communication medium, and... nope. almost unchanged since i looked at it last in June. Bluesky is a spectator platform where a small number of accounts receive most of the visibility and smaller accounts are effectively invisible. The introduction of new feed algorithms (to the degree that happened, there aren't really many that I can find in wide use) did not change that. This is a non-normative analysis: in some cases, it is good to have a medium that promotes some very small number of posts and accounts, eg. to surface singular events, etc.
From a 25h sample of the firehose...
600k posts, 2.4m likes, 250k boosts, 350k follows
40% of posts receive 0 likes, 70% receive <= 1
accounts in the 99th percentile of likes received 44% of likes, accounts in the 95th percentile received 74%
40% of posts were from accounts within the top 95th percentile of accounts by likes received.
the maximum number of likes for a post by an account not in the top 95% is 32.
The first plot below shows the cumulative sum of likes received on the y axis against each account in the sample on the x axis - this includes accounts that didnt' post during the sample (but would still have posts that could be liked, so this also shows the extreme recency bias). The second plot is a hockeystick showing the number of likes (not cumulative sum) received on the y axis per post on the x axis.
For background, the default algorithm only cares about likes, boosts don't matter, which is why i am calculating things by likes here - they are the primary algorithmic signal.
These are the same calculations that I did back in June, but this time i'm leaving the firehose open to do a longer sample to be able to parse momentary virality from persistent effects.
@marcosflobo I personally like it much more there than on Mastodon, but I'm not sure why, it's very subjective :)
The folks I followed on Twitter, iOS developers, are mostly here, but I've found some new friends there - mostly devs that build things on #ATProto. We have a small community there, on Discord and #Bluesky, we help each other, share tips and so on.
While I don't want to get into #ATProto until some questions on governance are answered (and I'd rather something like AP), there is a lot to like here and a lot of things I think we'd be extremely foolish not to use as concepts as we look to the future of #ActivityPub.
Going into some of the specific things I like a lot here that are simply not possible* in AP today…
(* yes yes it is possible to conceive of how to do it in AP in a completely non-interoperable way, you're very smart)
There's a strong separation here between the action and the model. There's something like a RPC sitting on top of the the model, so you don't see a conflation between the command and the underlying data structure.
This means that the system does not need to define the entire world and all use cases, it just needs to be able to express the things you might do and leave the underlying data to the specific application.
They have made it so that the lowest level of the system, equivalent to a server, is expected to go offline and made it so that accounts are portable between them.
This has a few advantages that are obvious, but one that may not be is that their architecture actually allows those lower level pieces to be very light weight.
I have some open questions on federation, but based on the patterns I'm seeing, I expect this to work relatively well when they finalize.
There is a strong "schema-first" way of thinking. This is incredibly powerful: they actually want it to be amenable to codegen and they want it to be easy to get started.
This is a sharp contrast with how AP works today.
Looking at the docs, I think most of these steps could be generated in most major languages.
That separation between the data object and the action means that it is easy to build, e.g., search on top of the system in a way that is interoperable. The schema-first approach also makes this very, very easy to work with.
You just need to define your application to have those primitives in its lexicon and you can add interoperable actions—not just activities—and communicate these out to others.
There's a lot to look at positively here and like. If we want something more grassroots—and more like AP—then irrespective of ATProto's ultimate governance we really need to look at some of the things ATProto is doing as we evolve into the future.
So it looks like most people aren't very interested in joining #Threads. I've looked at some tech profiles over the weekend, and some are active there, but not that many (though still more than on #Bluesky…). Definitely more are posting here.
Honestly, I'm now on 3 different Twitters, I don't feel very excited joining one more - and especially one that is completely centralized and has no public API at all, and from a company that's known for having very closed and limited APIs on their sites…
I don't know how things are going to develop, but I think an ideal outcome would be if it all somehow got connected in the end - #Threads federating with #ActivityPub as promised, #Bluesky / #ATProto getting connected with ActivityPub over some kind of bridges, and #Twitter… not sure, but somehow being connected in the end too (possibly after going bankrupt and taken over by someone else) 🤷🏻♂️
I really like what I'm seeing with #ATProto so far in terms of the communication and design layout, also with how thorough they are being in thinking through the auth and security models: https://atproto.com/blog/2023-protocol-roadmap
It's complicated and there's A Lot™ but it is neatly modularized.
Governance is my big open question mark and very, very important going forward, but from a design perspective I see a lot to like.
Something I think the #ActivityPub community needs to seriously consider here is:
How do we create a living standard? How do we avoid ossification and stagnation?
How do we create a situation where it is easy to federate and interoperate with others?
What features is #ATProto using that we can be involved with or interoperate with, even if we don't interoperate with the protocol itself? For example, can we support their use of DIDs and make that support easy to universalize?