Hi @evan@cwebber @pfefferle
Timely question: Does anyone know the status of a formal or informal building out an #ActivtyPub test suite, akin to the one used to test http://webmention.rocks but for Acitivity pub.
I know the activitypub.rocks effort ended & I see this one that seemed to evolve from that but has been derelict for a while.
After you do anything you post mentioning that account bounces to all others who follow it.
I add such groups into my own #mastodon list, and using #Ivory hide that list from my main feed so it keeps the convo in one list view.
See if that works. If folks hate, will find another venue
All of this super helpful: but def seems that a full Test Suite feels critical now to get together quickly, for the exact reason @pfefferle mentioned and that was mentioned in the w3 mailing list:
"If someone came along selling you an ‘activitypub’ app or service— or a ‘fediverse’ or social ‘web’ thingo— In an open market where people lie, cheat, or bend the truth, how would you or any of us try to figure out if it was compliant?"
I remember back in RSS days, there were a bunch of RSS checkers that would validate and say 'yes, this is valid RSS' but there was one exceptional one that could say 'Yeah, it's valid, but doing X would be better', and 'OK, but RSS client Y won't accept or see that tag'.
A test suite that validated base ActivityPub functionality/compliance would be great. One that understood common extensions - even better!
@OpinionatedGeek@tchambers@evan@cwebber@pfefferle@gabek Well, let's get the ball rolling then... What kinds of features do we need / want from a test suite? How would we be able to test Fediverse instances on it? How can we make sure it's useful as devs are creating new ActivityPub tooling?
@OpinionatedGeek@tchambers@evan@cwebber@pfefferle@gabek And, off the top of my head, those items would be: Digest signature authentication, Webfinger, Inbox, outbox, and other required endpoints, and confirm that messages of all types are properly formed.
@tchambers@OpinionatedGeek@evan@cwebber@pfefferle@gabek LOL, well it does bring us into the first debate for any project of this size - what language should we use? I haven;t taken the time to learn go yet, but I guess it's going to be a necessity at some point. Not sure how complete this application is, so not sure how much effort it would be to revive it.
@OpinionatedGeek@tchambers@evan@cwebber@pfefferle@gabek The real challenge is that a test suite is actually a reference implementation - it by definition becomes the standard that others are expected to follow because it enforces that standard. So... whomever builds it (and I'd be glad to contribute) needs to implement according to both the published spec and the ad hoc spec as created by Mastodon and Lemmy. Is there a canonical spec to follow these days, or is W3C it?
(I had a brief look at it and couldn't see an easy way to run it in a container, and I really didn't want to get into installing Guile/Scheme back then.)
Hi all, what @robz mentioned is exactly what I am building out. The only problem is it's the test suite I use to validate my own implementation of Fediverse logic in @nodebb.
@OpinionatedGeek@tchambers@evan@cwebber@pfefferle@gabek I can kind of kick it off. I'm building some tooling that exchanges ActibityPub messages. I need to test the secure digest signature part as well as ensure my messages are well formed. But I feel like these are separate things to test - I need to get the digest right to federate and pass messages, but I need messages to pass to test the digest. I'd like a tool that allows me to test each separately, then together.
Yes, I've been researching/designing a modular test suite that would layer out different groups/scripts of tests for different actors, first of AP then (optionally/extensibly) of specific supersets of AP. is there a repo I can watch or a mailing list I can join to coordinate so we don't all do redundant work?
I'll be creating a mastodon-compatible fediverse "group" we can all post to and see how that works... No one group repo....YET, tho we have found several older dead Github and Gitlab efforts that were abandoned that might be worth looking at what we can pull from those. All those in the thread here.
"Yeah, your actor endpoint is valid, except #Mastodon expects preferredUsername to never change (it's your unique identifier), and name to be the one that is in flux, despite the nomenclature"
@tchambers I think the bigger issues is what exactly you are trying to test against, because implementing "valid activitypub" doesn't mean you get compatibility with anything really. We basically need a test thats a bunch of "fake" mastodon, lemmy, pixelfed etc servers that can be have automated testing against.
@tchambers But what does having a "complete test suite" mean? None of the software we use in the fediverse is "complaint" to a single spec cause everyone is implementing a different set of features and are only partially compatible across the spectrum what is and is not the fediverse.
I agree we need a way to test compatibility, but I think the harder part is figuring out what is a sup-standard that all these things MUST implement to be compatible.
@tchambers@liaizon I struggled with that test suite (I'm trying to write an ActivityPub server of sorts).
What I wanted was something that would tell me 'yes, you're sending a valid ActivityPub response to event X' or 'no, that's not valid', or 'hmm, OK, that's valid but Mastodon expects Y'.
Instead that test suite starts a web site for you to interact with, requires you provide TLS certificates, and after all that I couldn't get it to accept my ActivityPub responses. It just sat waiting.
Unfortunately, I have not had any success with go-fed. I think it’s pretty much the standard in the Go world, but it’s far too unwieldy for me. @tchambers@liaizon
@liaizon@tchambers that is a really good point and it is the problem that costed me the most time. Getting a basic ActivityPub node interoperable with Mastodon, Pixelfed and others!
Add comment