Full text search has been merged in #Mastodonmain branch, and will be in the next (and final?) 4.2.0 beta 🎉
It is opt-in, so it will take some time to be filled with people content as they enable their profile to be indexed, but this was one of the most wanted Mastodon features for some time.
We plan to deploy it to mastodon.social and mastodon.online in the coming days to have a bit more feedback on it and see how it behaves in the wild.
To opt into your content being searchable, once your instance is upgraded to support this feature, head to the your profile, and the new “Privacy and reach" tag, then tick the "include public posts in search”.
We will try to ensure users know this setting exists, nothing 100% defined yet, but we may add a dismissible banner asking to review your privacy settings page.
@gpollara We plan to release 4.2.0 in the first half of september. For now, some servers are running the latest code to find out any issues and test the code.
@renchap a change like this is definitely worthy of a banner of some sort 👍 If you're worrying people might be annoyed by it, I wouldn't be. Better for people to be annoyed by one banner they have to dismiss than to be annoyed at never having been informed of such a big change.
@Brendanjones I personally think this is a useful feature to gently remind users to check their privacy settings regularly, especially when we make changes to them or add more options.
@renchap hmm I wouldn't personally do periodic reminders if there haven't been any changes to the settings, though I can see the argument for it.
People should be prompted to set their privacy prefs as part of the onboarding flow, meaning they should know where to look if they feel the need to update their settings later on. Definitely notify if settings have changed, though.
@renchap this looks like it is the case but can you confirm that toots posted as unlisted (like this one) will not get indexed? Are they indexed for self search?
@rysiek Not searcheable. This is opt-in for both existing and new accounts.
We will probably improve onboarding to point out the new privacy page and showing people they can chose how public they want their profile / content to be.
Full-text search allows you to search for words in the content of statuses (if your instance has ElasticSearch configured), compared to the current version which only allows searching for hashtags.
In addition, you get a whole lot of search modifiers for advanced search (no UI for those yet):
@LinuxAndYarn You can use -is:reply
For boosting it does not really make sense, as you are searching a post content. We could have boosted_by:account maybe, but this is not implemented right now
@frozencanuck Profiles are already indexed (in 4.2.0 at least), if your profiled is marked as “discoverable" (the “Feature profile and posts in discovery algorithms" setting).
If this setting is not set, then only the user name and display name are indexed.
@developit@renchap All posts get indexed regardless of content, but only if the poster decides that they're indexed.
I'm not 100% sure what you mean by the question (do you mean a non-indexable post, or an indexable post with no text?), but image-only posts will be searchable using has:image.
@mattswift@developit@renchap i take it they mean can use "has:image" with no keywords and find all posts with images, including those from accounts which are opted in for indexing?
(In other words is the indexing for text content separate from the indexing for search filters?)
@tokyo_0@OutOnTheMoors@renchap@SuitedUpDev it.. is opt-in from the start..? Well from the start of the full text search feature. It feels like it’s delivering what your after..?
@OutOnTheMoors This new feature is opt-in, so you do not need to opt out.
Your content will not be indexed unless you tick the new privacy setting to allow it.
@renchap When I search my @ on Google, my own posts don't immediately appear. But all searchable replies do, so it takes a flick of the finger to scroll up and find my posts.
This is exactly the same "protection" you're promising here, n'est-ce pas?
And regardless, when being opt-in causes it to fail and people start clamouring for opt-out what then?
Search shouldn't be supported because it is being demanded by people who want this to be like Twitter, when it's strength comes from not being Twitter. 🤦♂️
Mastodon should move away from not towards Twitter.
I want people to discover me, but through organic connections such as boosts by those they follow. That makes for organic, growth that is more about quality and connection.
Search is a different thing and will change the nature of connections, communities and Mastodon.
Other people want to use Mastodon differently though and their needs are just as valid.
The developers are trying to meet the requirements for a broad spectrum of people and find the right balance. That's not an easy thing to do but I think they're making a solid attempt.
@mackaj
Compromise isn't appropriate in all situations, such as when it moves the project towards features that were defined not for users benefit, but for profit
Because users want, or expect features they are used to elsewhere, perhaps without knowing or thinking about their impact, is not enough reason to accede to their requests. Let them choose a service that suits them rather than degrade one that works well and was built w different goals/values.
Twitter was built for profit and that's the reason Mastodon is and must stay different, and not try to mimic a system that was built for profit rather than users' benefit.
Perhaps when you consider that, and that there are different use cases, some benefiting from search and some hindered or degraded by it you will see the point I'm making.
Because something was useful on Twitter doesn't make it desirable here unless you want Twitter again.
I don't get it. There are no adverts here to drive a profit economy off the back of Mastodon. And I fail to see how improving the ability of users to find others who are enjoying the same game they are playing, or reading the same scientific paper is going to change that in any way.
@LizEllisPhD@markhughes@mackaj@OutOnTheMoors@renchap it’s an open platform. Fork the code, fire up your own server, set your own rules. You can still connect with the rest of the fediverse but do so on your own terms. It’s rather good.
@pee@renchap Yeah. I didn't opt-in to be searchable on Google, but here we are 😂
Funny how it's only Twitter and Mastodon that appear in Google searches, though. Almost like other platforms have thought this through a bit better.
@pee It becomes a minefield.
I have spent years investigating social media manipulation and how each feature can be used as an attack vector. Search is the easiest to automate.
@OutOnTheMoors@pee Could you please elaborate on how search enables manipulation and can be used as an attack vector (a link to an article or social media post is fine too)? I’m genuinely interested.
@ilyess@pee
On an "in-app" search (ie Mastodon), you can search words. Like "abortion" or "sneakers". You isolate an interest group and use your bots to follow (seeking followbacks). Then you either target-spam on their interests or volume-spam into their TLs.
The wider out-of-app search engine (such as Google) one is used to flood individual servers with requests - to create a DDoS effect.
Or, again, volume-spam can be used.
The effect of both methods is they create the need for more automated ways of handling things like traffic and admin overload.
I haven't even got to the troll-direction aspects, but you should already be able to see the problems.
@OutOnTheMoors@renchap we need to all be mindful that we’re posting publicly. Public web content gets indexed and scraped.
If you don’t want what your posting to be see at all by anyone other than the intended recipient then Mastodon is not the right platform to use. Most social media wouldn’t be.
@ladyteruki This feature works with an "indexable" flag set on your profile.
This flag is federated, so if you change it, Mastodon will broadcast the change to instances known to your profile.
Instances will now also refresh remote profiles every 7 days (if they receive content from you), to ensure the flag propagates correctly.
@renchap : actually while I'm here, may I ask another question ? If I want to reply to a toot posted by a mutual, that I saw a couple days ago (and assuming this person has opted in), can I search among toots specifically posted by that mutual ? Or will I have to scroll through aaaaaall the existing toots with the word or phrase that I remember ?
@renchap Question, is there a modifier that only returns posts that you've interacted with, like with the current behavior? I know a few people that like that functionality and it would be somewhat disappointing if that was completely lost.
@esm I am not sure, I remember it being discussed but I am not sure if the old behaviour has been kept somehow.
An issue with this feature is that we need to store who interacted with which post in ES, and this might create huge amount of data.
I added the question to my list of things to clarify!
@renchap hmmm, that does seem like an issue, hopefully something that's addressable though (if ES doesn't instantly break, of course). thanks for the response!
@esm This is only the first step of the implementation and I know people will definitely continue to work on improving it, so I can only assume it will be implemented at some point.
IIRC one of the issue is that for very popular posts, if you store user ids of every interaction in the document, you can get into trouble and reach the max document size
maybe it's time to look again at ES alternatives? I've been wanting to ask for a while what exactly the blocker is that ES has that the competition doesn't, considering that some other software I've worked with that has similar scope has moved to stuff like meilisearch without issues (apologies if this is a bit off topic)
@esm About Meilisearch specifically, it does not support clustering/HA. This is a must have, for now the index sizes are manageable on a single server (50-80 GB), but we did not index much. We do not know yet how many people will opt into the new feature and how much data this will use.
What I described is what I remember of the discussions on the topic, but I think this can be solved in multiple ways. And I think we are open to alternative implementations (like Postgres FTS) for smaller servers
@bart@fuzzychef The current search implementation relies on the Chewy Ruby gem, which uses Elastic's Ruby client under the hood. So we "officially" only support ES, but I know some instances are using OpenSearch without any issues.
@renchap@bart Oh, right, Elastic has licensing issues too.
I was just thinking of overhead. Postgres has FTS capabilities, so being able to use that would make FTS much more accessible to smaller instances, since they wouldn't need to run a third DB service. Opensearch doesn't really help with that.
@morethanevil not from us. I wonder if we could support multiple backends, but thise 2 do not support HA/clustering and this is needed once you get to a certain scale.
@renchap That's fantastic news, especially search operators. 🤩 Is it possible to add an operator similar to Gmail's from:me instead of from:@katzentratschen? It's kinda bulky to type in your handle, but even more so to type in your handle plus your instance.
@renchap Thank you. 🤗 Speaking of the ideas list: Is there a proper way to give feedback apart from GitHub? Preferably within the Fedi, e.g. a hashtag.
@renchap Very nice. before: and after: are clearer than ex-birdsite's until: and since:.
One issue I ran into with their version is: timezones. When I needed exact day boundaries for a search, I used the until_time: and since_time: operators, which take a Unix timestamp. Have you thought about adding those too?
@viktor@OpenSearchProject This is relying on the Chewy Ruby library, which uses Elastic’s Ruby library. It does not officially support OpenSearch, but I hear people have been running OpenSearch + Mastodon without issues
@renchap@OpenSearchProject Thanks. It's better to use open source tools rather than source-available like Elastic 😀 I wonder if Mastodon hosting providers will be able to offer Elastic to their customers with a source-available license since they would be competing with Elastic.
@karussell The current search implementation relies on the Chewy Ruby gem, which uses Elastic's Ruby client under the hood.
But I know that some instances are running it against OpenSearch without issues.
Merging this PR is only the first step, and we know there are probably bugs, things that do not work as expected, or missing features. This is not the end of the journey and there will be improvements! Please open Github issues if you see any problem related to full text search (once your instance has it)
Also, multiple people asked if alt texts are indexed, and I can confirm they are 👍
@renchap@vmstan Great to see progress on this! Thanks.
Reading through the doc, I’m unclear if this search supports my 90% use case: searching all posts I’ve seen in my timeline recently (some will be from people I follow or who people I follow boosted that are on other instances).
If this isn’t easy and supported then this misses the “oh, I just saw a post about that recently, now where was it?” use case which is a pretty big miss.
@renchap@vmstan Thx for the heads-up.
I am mainly only interested in my posts being searchable by me ONLY (rather than publicly searchable by others), which the article above seems to say is an optional setting per instance?
If that's the case, how can I tell whether an instance has this "own posts" option enabled (other than trying it +seeing if it works)?
...and what version is this "own posts" search available from (v4.2.0 also)?
Thx again.
@renchap I would love an operator similar to from:account but returning posts boosted by the specified user. This is a feature Twitter never had and its absence always bugged me! (I suppose federation makes to:account hard to implement; Its main use on Twitter was to check if someone has already written a reply similar to what you have in mind.)
@renchap what does is:sensitive mean? it is has:contentwarning?
(talking about language, can I identify in the web UI which language a post is tagged? Ideally I would appreciate to see it when selecting a post, in its bottom line near timestamp, visibility, client...
I am asking because I have the suspicion that some people/clients set it wrong, I have one person when replying in German to a German post, I have to switch from en to de every time, which makes me suspect she sets it wrong)
@renchap Is it planned and/or possible yet to add modifiers like is:starred or is:bookmark? It's quite difficult to find the public posts I've faved or saved within a large list of other posts matching a search string.
@katzentratschen Yup, it kind of does what you asked, but it includes everything you faved/replied to/boosted (ie, what we consider as "interacted" by you)
Making it more granular is more difficult, as it would require storing the exact relationship (why you interacted with it)
@renchap I've noticed that. But it's still waaaaay better than having no filter at all. And it's possible to exclude replies with in:library -from:katzentratschen. (An option to save frequent searches similar to Gmail's retired labs feature Quick Links would be superb.)
@renchap How do you make sure that bad actors don't misuse this? For example; I understand that only posts from users with the indexable flag are indexed. Isn't it easy for bad actors to add the indexable flag to every account on their side? So that all posts will be indexed after all?
@joenepraat There is no way to prevent external people to index your posts. They can do it right now if they want, as posts are public information (except private mentions) and can be fetched / scraped from your instance.
If an instance is ignoring the setting by changing the code, your instance admin can defederate them to prevent your content to be federated there (but they could still scrape your content if they want…)
There is no real technical solution to this.
@renchap I understand. But I'm afraid those illegal scrapers are not easy detectable. Right now, a scraper is better detectable because of the amount of IP requests.
@renchap Another question. Will this not add a huge load on Electricsearch? Electricsearch hasn't great performance in my experience, so are you not afraid it will costs especially smaller instance to much performance?
@joenepraat It will add load, but for now we do not really know how much. This is why we plan to deploy this on mastodon.online and mastodon.social very soon, so we can get a better idea of what kind of load this will generate.
From my experience, ElasticSearch scales quite well if it is correctly configured.
@renchap OK. We will see. The problem is, is that mastodon.social and mastodon.online are larger servers, and I especially see problems with smaller servers.
@renchap Renaud I’ve just noticed that the new privacy settings are under the profile settings page. Privacy should really be a top level setting!
Also, many of these privacy settings don’t have anything to do with your profile so putting the privacy page under profile is not such a logical location.
@renchap “Include public posts in search results” is going to be a separate setting to “feature profile and posts in discovery algorithms”. Also separate to “a profile page in search engines”.
@renchap how does this propagate across instances, or interoperate with other ActivityPub servers?
From a user perspective, I expect that the search will include my timeline, or basically all posts known to my instance. But the users who posted them may be on a different instance. So my server has to know whether it is allowed to index posts from that user, right?
Is there a search parameter to only search my timeline? I often want to reply to a post I have seen, currently finding it is hard.
@KarlE Your instance will only index content it knows about, either because a user follows it, or it comes from a relay. Only content from account that opted into being indexable will be indexed for full text search.
There is nothing yet to search in your timeline only, but this is in the list of things we would like to add (if technically possible)
@nicod car ça vient d’être mergé, donc ça sera dans la beta3 😉 Semaine prochaine si tout va bien.
La recherche full-text actuelle ne cherche dans dans les statuts avec lesquels tu as interagi (et les hashtags)
@nicod c’est un truc qu’on va changer, actuellement quand tu tournes directement depuis git ça utilise le dernier numéro de version publié. Oisaur tourne directement sur la branche main, yolo (mais ne faites pas ça sans suivre de très près le repo github, moi c’est pour tester en prod)
@renchap@mmasnick The main reason, why people don‘t use Mastodon is now an opt-in feature. Apart from about 7 nerds nobody will know how to activate this and the main reason why people don‘t use Mastodon will persist.
Bravo!
@renchap Such a smart, thoughtful approach. As potentially inconvenient as requiring a “yes, please” action feels, it’s exactly the kind of protective behavior that reminds me that the fediverse is about people.
@aguamielerogpz Your instance will index the content it knows about (followed by instance users, or from a relay), from accounts that marked themselves as wanted to be indexed.
@renchap Does opting out, after having opted in for a while, remove all previously indexed posts from the index? In other words, if I opt out, would that make all my posts, including those that were previously indexed, no longer full-text searchable?
@renchap Renaud, are there any preliminary results as to how much load the search features add to the #Mastodon servers?
Is this a mild overhead or more substantial?
@renchap thank you that is amazing. May I ask why not having the search/indexed by default for everyone instead of having to opt-in? I am just wondering if it means some contents will still be hard to find later until we ask the given account to opt in. It is actually maybe not that bad to require opt in to avoid indexing everything I’m the first place but I am just wondering your initial thoughts to have that opt-in?
@renchap Who hosts the indexes for this? This seems like a project that is a giant data storage problem, and I haven't read deeply into any of this, but who pays for storing the index?
@renchap Thank you and the whole team! Search is basically the only feature that Xitter has that works well still that is missing here (arguably quote toots as well, but that is contested and you can still do it here if you try hard). Having search means that I can finally import my great threads fr on Twitter, repost them, and have them be useful to people again! 🙏✌️
Add comment