supakaity, 1 year ago

Preface: This is a long, technical post so I apologise in advance. Please avert your eyes if such things offend you.

Hi everybeings!

Overview

@ada and I have discussed many areas where we see problems in the way the Fediverse currently works and where would like to see improvements made to improve safety of the fediverse.

Safety on the fediverse is one of the topics we hold most dear.

One of these areas (and one that has recently garnered a lot of attention) has been in the area of full-text search, and despite the option for account-wide no-crawl options in many pieces of fediverse softwares, this option is not-federated, non-specific, non-granular and not-for-purpose with regards to fediverse searching. It's specifically designed and worded for crawler bots at a html scraping level, and while we could repurpose it for fedi-searching, it just doesn't feel right or quite fit.

There's been a lot of talk on a lot of different levels out there, but nobody's come to a concensus. A lot of people are talking about how we should do it and what standards we should use, and generally making the whole concept a lot more complicated than it needs to be.

In this post, I want to share with you some of the ideas that I've gathered through my research, and implemented in a way that is simple enough (KISS), not overly complicated (YAGNI) yet still fit for the purpose I need as both a software engineer and an instance admin, and that anyone accessing the content will need.

The idea in sharing this is not to convince me that it shouldn't be done, or to create the absolutely most perfect solution possible that's going to take 20 years to build and will be outdated by the time we get there.

This will be getting built and going out in weeks, not months, not next year. It's needed now. This is my current plan how to implement something we needed months ago, now. This is your chance to change my mind and help refine this plan before I start really coding it up.

Technical/implementation details

A lot of these options will be settable as a default value in your settings as well as at an individual post / file level during composition or afterwards at editing stage.

In terms of inbound federated AP objects which do not support these new fields, we will try to infer the intent based off existing AP fields and other metadata when present.

Specifically, we will be using the Mastodon Account Lookup API (GET /api/v1/accounts/lookup?acct=) to get and store the noindex flag during actor creation/refresh. This will also allow us to put the noindex meta tag on any HTML pages containing that actor's posts.

Searchability

So you don't want people to be able to search for you. But is that everybody? Maybe you want your followers to be able to find your posts? Maybe the people you mention should be able to find your post? What about actual users on your local instance? Maybe a particular post you never want to appear in searches? Maybe a particular post you write contains stuff you want everyone to see?

We will be adding a global default and per-post override federated in AP as hk:searchableBy which will be settable to public, private or a combination of followers, mentions or local. This is different to the currentvisibility in that you can have a public visibility post that is searchable by followers and mentions for example.

{
    ...
    "searchableBy": ["followers", "mentions"]
}

Interactivity

The other problem is that at the moment you can't control who can reply, boost, vote and/or react to your posts.

To this end, we will be adding per-post overrides federated in AP as hk:canReply and hk:canInteract which will allow you to specify like on searches a combination of public, private, followers, mentions and local for replying to, and boost, voting or reacting to your posts respectively.

{
    ...
    "canReply": ["mentions"],
    "canInteract": ["public"]
}

Licensing

Licensing and attribution can get pretty tricky. On the fediverse, we're assuming that if you're making a post, then you have the rights and willingness to let the content be federated. If you didn't you wouldn't post.

However posting content sometimes requires you to provide the license and attribution that you're using along with the content. At the moment there's no reasonable or standardised way to provide this information on an image or in a post, thus you're in breach if you post a CC-BY licensed image.

So to alleviate this, we will be providing 2 federated AP fields hk:licensing and hk:attribution. The licensing field will contain the URL of the license under which the content is being shared and the attribution will contain any links to the source content/s and/or creator/s.

{
    ...
    "license": "https://creativecommons.org/licenses/by-sa-nc/4.0/",
    "attribution": [
        "https://github.com/supakaity/icons"
    ]
}

Crawlability

So crawlability is about indexing of the content by external parties whether they be search engines or fedicrawlers. It will provide a per-object level flag under the AP field hk:crawlable that lists whether the item may be processed by a bot. The possible values are true or false.

{
    ...
    "crawlable": false
}

Quotability

Now this is a controvertial one. Sometimes people out there don't mind their posts being boosted, but don't want them to be quoted. You know what we say? If you don't want your posts quoted, we should respect that and not allow people to quote you. So we'll be supplying an AP field hk:canQuote that lists who can quote your posts using the same combination of actor types as above.

{
    ...
    "canQuote": ["followers"]
}