Migrating a list of Reddit posts to Lemmy

cross-posted from: https://lemmy.fromshado.ws/post/793

Disclaimer

You are responsible of cloning whatever content you decide to migrate. I suggest to keep it to posts you made originally or have been given permissions to migrate. DO NOT SPAM

Background

Following all the recent issues I caught wind of Lemmy, I was aware of the Fediverse already and been present of Mastodon for years but unlike with microblogging alternatives I don't mind losing contact with people on link aggregators.

But since I was considering also deleting or overwriting all my original submissions (lol it's all gacha memes) I wanted to keep them available somewhere else. So I migrated a bunch of them to /c/fireemblemheroes

The program

So I coded a small python script (it's actually 500 poorly written lines) that given a file where each line is an URl of a Reddit post it can parse the corresponding content and using Lemmy's API clone them into your instance.

https://lemmy.fromshado.ws/pictrs/image/d13f5f03-6fb1-4eed-8971-ea24da3d0000.png

sample execution migrating 40 posts and their comments into a local test instance

Migration of text

For the most part simply copying and pasting original and texts back into Lemmy works.

Still there's a need to clean the body to make sure inlined picture links are expanded and characters are unescaped.

Migration of media

The post might be a single link or might contain links hosted on Reddit in it's body.

If enabled, the script is capable of downloading this media and reuploading to the pictrs instance asociated with it, then replacing with the new selfhosted link. Otherwise the original link is kept.

https://lemmy.fromshado.ws/pictrs/image/c56a4464-df35-4e55-ab37-b5a834654db5.png

I suggest to disable video uploading though, as most of the time pictrs will not finish handling of the file before lemmy decides to timeout the connection

Migration of comments

The script is capable of tracing all comments in the particular post and migrate them while keeping the threaded relation between parent and child comments.

It also keeps a credit of name and date to the original author. In any case migrating comments will make you hit rate limits severely, increase runtime drastically.

This option really is only intended if you are migrating your community from reddit and want to keep all the top content.

https://lemmy.fromshado.ws/pictrs/image/02fb3ebd-25c8-4522-b2f7-717a1356a36f.png

This is how an URL link looks with migrated thread comments

Migration of upvotes

This is not possible without possible affecting federation. You would require editing the database directly as obviously the API doesn't allow it.

The same happens with original posting dates.

Running it

The prerequisite is having an user to authenticate to the instance you will use and the community where the posts will be migrated to exist already. The code is located at https://github.com/Eskuero/antenna2lemmy

Clone it locally:

$ git clone https://github.com/Eskuero/antenna2lemmy; cd antenna2lemmy

Create a python virtual environment and install the required dependencies on it:

$ virtualenv env; . env/bin/activate; pip install -r requirements.txt

READ THE CONFIGURATION FILE CAREFULLY, it documents each option. Adjust it to your needs.

Execute with the following command, where "personal" is the name of the community where the posts will be migrated to and "links.txt" a simple text file containing a single Reddit url per line. They are comma separated.

$ python antenna2lemmy.py personal,links.txt

The program should start running with a curses interface to report on progress. You can disable it by passing the environment variable DEBUGMODE=1.

The full log output will be saved to migration.log on execution directory

Reminder to be respectful of the other users of your instance, just because the tool runs automated and you are essentially crossposting from Reddit spamming a lot of migrated links might get you banned