Introducing Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration

I’m excited to announce the first alpha preview of this project that I’ve been working on for the past 4 months. I’m initially posting about this in a few small communities, and hoping to get some input from early adopters and beta testers.

What is a DHT crawler?

The DHT crawler is Bitmagnet’s killer feature that (currently) makes it unique. Well, almost unique, read on…

So what is it? You might be aware that you can enable DHT in your BitTorrent client, and that this allows you find peers who are announcing a torrent’s hash to a Distributed Hash Table (DHT), rather than to a centralized tracker. DHT’s lesser known feature is that it allows you to crawl the info hashes it knows about. This is how Bitmagnet’s DHT crawler works works - it crawls the DHT network, requesting metadata about each info hash it discovers. It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed.

This means that Bitmagnet is not reliant on any external trackers or torrent indexers. It’s a self-contained, self-hosted torrent indexer, connected via the DHT to a global network of peers and constantly discovering new content.

The DHT crawler is not quite unique to Bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. However that project is no longer maintained, and does not provide the other features such as content classification, and integration with other software in the ecosystem, that greatly improve usability.

Currently implemented features of Bitmagnet:

  • A DHT crawler
  • A generic BitTorrent indexer: Bitmagnet can index torrents from any source, not only the DHT network - currently this is only possible via the /import endpoint; more user-friendly methods are in the pipeline
  • A content classifier that can currently identify movie and television content, along with key related attributes such as language, resolution, source (BluRay, webrip etc.) and enriches this with data from The Movie Database
  • An import facility for ingesting torrents from any source, for example the RARBG backup
  • A torrent search engine
  • A GraphQL API: currently this provides a single search query; there is also an embedded GraphQL playground at /graphql
  • A web user interface implemented in Angular: currently this is a simple single-page application providing a user interface for search queries via the GraphQL API
  • A Torznab-compatible endpoint for integration with the Serverr stack

Interested?

If this project interests you then I’d really appreciate your input:

  • How did you get along with following the documentation and installation instructions? Were there any pain points?
  • There’s a roadmap of high-priority features on the website - what do you see as the highest priority for near-term development?
  • If you’re a developer, are you interested in contributing to the project?

Thanks for your attention. If you’re interested in this project and would like to help it gain momentum then please give it a star on GitHub, and expect further updates soon!

Willdrick,

This looks kinda neat, I even tore down my whole servarr stack to give it a go, alas I can’t get bitmagnet to “talk” with prowlarr. I’m probably doing something really stupid, but I can’t figure out how to add the whole thing under a single docker network, I get errors like network somename was found but has incorrect label com.docker.compose.network set to “”

mctoasterson,

Maybe I’m misunderstanding but wouldn’t it just be easier to use a good private tracker, assuming you can get an invite?

lud,

Yes, of course.

Molecular0079,

Is it safe to run this without a VPN if I am just using it to index?

pedroapero,

Great project !

Naming conventions are missing some important information like bitrate, color depth, and most importantly language and subtitles.

Do you plan to scrape additional infos from known torrent sites (searching for torrent hashes for well named torrents) ?

mgdigital,

Scraping torrent sites will be avoided is it’ll be prohibitively slow and break the self-sufficiency concept - we’ll infer as much as possible from the torrent meta info alone. You could have a guess at the bitrate from the file sizes. Sonarr/Radarr will already do this for you with quality profiles I think.

LienNoir,

Hi, am i missing something, the bitmagnet image keep restarting when i check with “docker ps”, the other 2 containers are working as intended. And port 3333 doesn’t show anything.

emhl,

What are your logs showing? docker logs -f bitmagnet

LienNoir,

log: exec /bitmagnet: exec format error

I am on ARM (pi4) maybe it’s the issue

drugo,

Yeah, that’s the error you get when trying to run an x86 program on ARM or vice versa

emhl,

the parent image should support that arm version, so you could just build the docker image locally on your pi and use that.

Btw. There already is an open pull request to add arm support

LienNoir,

thanks, for pointing that out, it works great now.

mgdigital,

There’s a PR currently open for multi-platform builds so should have this sorted soon

Dasnap, (edited )
@Dasnap@lemmy.world avatar

It’s only once you install something like this that you realize just how many torrents are porno.

I’ve always been curious about ‘Anal Police Stories 2’ but I’ve never found the time.

Salamendacious,
@Salamendacious@lemmy.world avatar

scrubs YouTube clip

spiritedpause,

Dude this is amazing! Exactly the sort of thing I’ve been hoping would pop up to further “decentralize” the torrent search experience.

So I’m trying to run it on my machine through the docker-compose option, and I’m seeing something weird. It shows as successfully running, but when I go to the port it should be running on, I get “unable to connect” on my browser.

When I check my containers running, it shows the 3 bitmagnet containers, but the port doesn’t show.

i.imgur.com/D4R1Le5.png

mgdigital,

Hi, the default port is 3333, which should be exposed if you’re using the example configuration here: bitmagnet.io/setup/installation.html - I’m not sure what the app is in your screenshot but the provided config definitely exposes that port and is tested on Docker for Mac.

spiritedpause,

Just pulled the latest and tried again, and it works now! Thanks

BlueEther,
@BlueEther@no.lastname.nz avatar

seems to work well

just one question, is it expected to have 10,000 out of 12,000 as unknown?

mgdigital,

Hi, yep that’s expected. Torrents will only move out of “Unknown” once the classifier is able to categorise them. The classifier currently only supports movie and TV show content, and can recognise these with quite high accuracy assuming a well-named torrent (and a badly named torrent is unlikely to be a high quality release). The other content types (music, games etc) can currently only be populated via an import (see the tutorial on the website). A priority feature is classifiers for other content types - however we will likely always have a lot of torrents ending up in “Unknown” given the poor naming of many crawled items. Another roadmap feature, smart deletion, could help in future with getting rid of all the rubbish whose contents cannot be inferred from the torrent name.

droopy4096,

@mgdigital, first thing I’be noticed: reliance on “heavier” database stack (pg + redis), at least from the first glance at docker-compose. My suggestion would be to have an option for minimalist setup with sqlite and without redis if possible. That would work better for those of us flying with minimal hardware (rpi, old PC and such).

mgdigital,

Hi, this is a great point and one that I’ve already given consideration to. I’ll address separately the issue of the primary datastore ,i.e. Postgres, and the Redis dependency:

Postgres as the only option for the data store

There are 2 reasons for this:

  • Performance: while SQLite could offer a simpler/embedded data store, it simply doesn’t have the performance and features of Postgres. Bitmagnet has a faceted search engine and is write-intensive (it will be discovering ~5k torrents per hour and writing these to the database along with associated metadata). As such, its database may not be suitable for running on older hardware. A SQLite adapter, if it was developed, may simply not be up to the job (although as I haven’t attempted this I can’t say what the performance would be like). That said, Bitmagnet itself is not especially resource intensive, you could probably run it on a Raspberry PI but point it to a Postgres instance on some more powerful hardware. At this stage I’ve only been running it on a M2 Mac Mini with Postgres located on its SSD and so would be interested to know people’s mileage on other hardware.
  • Development, support and maintenance overhead: I’m a lone developer and this project is already too big for one person. A SQLite adapter, if feasible performance-wise, I think could only happen if other contributors joined the project as my to-do list is already pretty long. It would have to achieve feature parity with the Postgres implementation which makes use of several Postgres-specific features and extensions. It would also mean a longer testing cycle and therefore probably a slower release cadence. That said, if there was enough demand and assistance then I’d be open to looking into the feasibility of this once the rest of the application is a little more mature and the current database schema more finalised.

Redis dependency

Redis is currently used only for the asynchronous task queue. I would like to have put this in Postgres, but there simply is not a good out-of-the-box solution that works well with Postgres and GoLang, and is actively maintained. I looked at quite a few queuing libraries and eventually settled on asynq (github.com/hibiken/asynq), which is a great library and does the job well - but could really do with support for non-Redis backends.

Using Redis here was a pragmatic decision that allowed me to make progress, rather than an optimal one. I guess I could have built a simple Postgres-based queue myself but that would have been a distraction and probably sub-optimal compared with a mature/separately developed library. It remains an option. Since I looked into this a new project has sprung up which I’m keeping an eye on - www.tork.run - it has a Postgres backend and looks like it might be up to the job, but is very new.

So yes, I’m very aware that the additional Redis dependency is not ideal and it may well disappear at some point.

mlunar,
@mlunar@lemmy.world avatar

Hi, those points are certainly valid and I have nothing against these picks!

I just wanted to chime in that perf might not be as big of a problem as you might expect. 5k/hour is 1.4/sec, which sqlite should for sure be able to handle.

In fact, you can do hundreds to thousands of writes/sec, as long as you batch them in transactions (as by default each query is executed in its own transaction).

droopy4096,

thank you for such a detailed response. I would love to contribute however at the moment my capacities are rather limited but otherwise I’d be willing to add sqlite adapter. From your description it sounds like currently architecture is narrowly locked on PostgreSQL features. In my daily job I love PostgreSQL for big apps and stacks but I’m also aware how “hungry” PG can be, which is why I’m wondering whether it’s “too big of a hammer” for this particular problem. Also, setting up single service is easier to novices vs maintaining several. Docker compose is nice but it has it’s limitations.

Stephen304,

A dht crawler is inherently an intensive service to run, magnetico used sqlite and would take 10 minutes just to load the splash page that includes the total count of discovered torrents.

Shdwdrgn,

Looks like a fun project, but will you be providing any info on setting it up from scratch? I just don’t have an interest in docker containers.

cyberpunk007,

Out of curiosity, why not? I’ve come around.

Dasnap,
@Dasnap@lemmy.world avatar

I personally love containers (probably because I use them for work) but I can understand someone not wanting another layer of abstraction if they’ve worked bare-metal for a long time.

Shdwdrgn,

I’ve just always used VMs for everything and set up each service to match my existing system. For example, my postfix servers have to all tie in to LDAP, mailman, and the host of services for authenticating email. It seems like the point of docker is to just have a completely preconfigured and self-contained setup. I guess I Just don’t see how that would work in my environment where I already have some services like databases or LDAP already running elsewhere, and I run multiple instances for redundancy. And if I have to reconfigure all that stuff in docker anyway, how is that any better than simply using my existing VMs?

cyberpunk007,

Used to be like you, then I moved from truenas core to scale where it’s now Linux and docker instead of freebsd and iocage jails.

So docker has this concept of persistent volumes. You configure all your settings in the initial setup command (docker compose) and define persistent volumes. This way you don’t lose your data.

Here’s an example, Plex. I run Plex in docker now. So my config directory is defined as a persistent volume. If I need to update Plex, or rebuild it or whatever, the container just updates and has all the data I need via the persistent volume. If the install is messed up or whatever I just get a newer image and run the docker compose and it fires up and mounts the persistent volume and off I go.

Basically it takes away the burden of having to figure out the OS configuration. Makes backups easier - and smaller. And the things are spun up, installed, and usable in seconds.

Shdwdrgn,

Not sure the OS configuration is really a burden :-) I have several servers I have to keep up to date anyway. And backups aren’t really an issue, I just run rdiff-backup on everything to provide a year’s worth of incremental backups, which doesn’t really take much extra space. Maybe one of these days when I catch up on other projects I’ll look into it though.

cyberpunk007,

On truenas scale though it’s just tiles in a web browser, it’s super easy. And since it runs on ZFS backups are easier too. Just click your way through periodic volume snapshot tasks.

Definitely a bit of a learning curve but it’s a sleek setup once you understand.

Shdwdrgn,

I’m not quite sure what “truenas” is? All of my stuff is individually installed, I decided a long time ago to split it up onto VMs that each perform an specific task. I have a main file server that runs zfs, then two servers to run the redundant VMs. There’s not really anything difficult about backups, I just add a cron job to run a script once a day and never touch it again, so I have backups of each VM but then the backups of the main servers includes the VM image files so each VM gets backed up twice. There’s a lot of info there but the backups of all the critical stuff only use about 6TB (I could actually cut that in half if I got rid of the backups from older machines).

So lets say I put in the time to learn how docker works, and then put in a lot more time converting all of my existing systems over to docker images… What exactly what I get out of all that effort? The thing that nobody’s been able to sell me on so far is that I don’t see how docker is going to make anything any easier, it just seems like it’s a “different” way to do things but nothing more.

cyberpunk007,

Your data footprint would be less. Maintenance is a breeze. If you update your image and it breaks, just roll it back. Less consumption of resources. No need to divide your storage and ram for VMs. There are millions of docker images so you can start something new in seconds. And the learning curve isn’t too bad if you’re on truenas scale. Truenas core is a NAS operating system built on freebsd (Unix), and truenas scale is built on Linux. Both use ZFS for the underlying storage.

Shdwdrgn,

OK so my current strategy is that when I want to do a major update I simply make a copy of the vm image file, then I can drop it back in place if something goes wrong. I run KVM which means it just gives out CPU and memory as needed even though I can set maximums. The resources I’m using are laughably small anyway, half the systems run fine on a single cpu core although it was nice to recently bump web and mail services up higher (I just upgraded over the Summer from Poweredge 860 servers to some R620’s – crazy difference in available resources!). Same with memory, I have some systems running on as little as 512M, but I just bumped my web servers up to 8G to give them plenty of room. Considering I have 64G in each server with tons of space for growth, I’m not worried about any of that. And storage space… well it seems linux is suffering with bloat since the introduction of systemd as I’ve had to increase my image files from 4G to 8G for updates, but it’s still a drop in the bucket for storage. And all the services use shared storage space for things like email and websites, and I have around 105TB of shared storage, so again not really a concern.

Now it sounds like I kind of need TrueNas to easily use docker, which means another system that I would need to learn from scratch? Truenas scale says it’s built on debian and yet there are no debian packages available to install it, so I can only assume that I would have to completely replace all of my existing servers with brand new systems that I have no knowledge of troubleshooting, just so I can replace all of my existing VMs with docker images which I also have no knowledge of how to troubleshoot.

Sorry but none of this is selling me on the idea - it just sounds like I’m supposed to replace systems that work perfectly well with new systems that I can’t fix when they break? I’m really not understanding where the advantage is.

cyberpunk007,

Not trying to sell you on it, you do what works best for you. Truenas scale is an operating system built on Debian. There will be no packages for it. It’s hard to explain until you start using it. I came from VMs on truenas core for many years and it was annoying to migrate to docker but after I used it for a while I liked it a lot more. It’s hard to explain without just using it, so if you’re not into playing around and what you have works great, then great. I’ve been working with jails and VMs and containers for well over 15 years since I work in IT so I’ve played with big and small systems. There are definitely some annoyances when it comes to the VM approach.

Shdwdrgn,

No worries, and thanks for all the info. Eventually I’ll have some time to look into docker, but for now it’s just an aggravation to me when it used to be a simple matter of just running a couple commands to compile something.

mgdigital,

Hi, yes this is mentioned on the installation page of the website, below the Docker instructions. The app can be installed Dockerless using go install; if you choose this option you’ll have to provide and configure Postgres and Redis instances for the app to connect to. That said, Docker is the recommended and easiest option.

Shdwdrgn,

I saw that, but didn’t recognize the ‘go’ command as anything available on Debian. Just did some quick digging though and now I see it’s a new language and I believe I have an idea how to get it installed for compiling so I will give that a shot.

paris,

Golang v1.0 was released in March of 2012. Not sure I would consider it a new language.

Shdwdrgn,

Oh interesting… I thought I read something that said 2017. No worries, I’ll get it figured out now that I understand what it is.

palitu,

Very cool!

prim3r,

This looks really cool! How resource intensive is this? What sort of storage requirements are there for this to be a reasonably reliable method of acquiring media? I’m probably just gonna find out myself. I’ve recently fully switched over to usenet, but this could make torrents pretty compelling again.

kautau, (edited )

As someone interested in Usenet, what’s the best provider and client to start with in your opinion?

prim3r,

I’ve been using easynews/nzbgeek/nzbget with an arr stack on debian and it’s worked well for me. I’m fairly new to usenet, so take this with a giant grain of salt.

kautau,

Cool, thanks for the reply!

Kushan,
@Kushan@lemmy.world avatar

Sabnzbd is probably the best choice of download client, fyi.

CosmicApe,
CosmicApe avatar

Linux program names are fucking wild

mgdigital,

Hi, and thanks!

As a priority I’d like to gather some more rigorous performance benchmarks, but I can give you some hand-wavey stats now: Bitmagnet is currently fluctuating between 2-10% CPU usage on my M2 Mac Mini, and is using ~120MB of memory having currently been running for around 48 hours. Overall, the GoLang implementation seems pretty efficient to me considering how much I know is going on in the background.

Disk space usage of the database- this will be highly dependent on 2 configuration options, the first of which I’ve only just added in the just-released version. Copied from the configuration page of the website:

  • dht_crawler.save_files (default: true): If true, file metadata from the DHT crawler will be saved to the database. This provides more rich information about a torrent, but will use a lot more disk space. If disk space is at a premium you may want to consider disabling this.
  • dht_crawler.save_pieces (default: false): If true, the DHT crawler will save the pieces bytes from the torrent metadata. The pieces take up quite a lot of space, and aren’t currently very useful, but they may be used by future features.

For me, 24 hours of crawling uses ~2.5GB of database disk space for metadata on the ~120k torrents it has discovered. Yep, that sounds like a lot, however 90% of that is taken up with the files metadata, and could have been saved by setting dht_crawler.save_files to false. In fact I may set this to false by default and allow users to opt-in to the full-fat torrent info.

I’ve also imported the entire RARBG backup (the SQLite one, see tutorial on the Bitmagnet website). This, along with all the associated metadata from TMDB, took around 4GB of database space, which seems quite acceptable considering it’s basically every movie and TV show. Note that this does NOT include the metadata on individual files as I described above.

A priority feature for me (detailed on website) is smart deletion - this would allow you to automatically discard a lot of data that can be automatically determined of no interest and therefore greatly reduce disk space demands.

deafboy,
@deafboy@lemmy.world avatar

Running for 6 days, save_pieces: false

My database is currently 184 GB

snekerpimp,

That was easy to set up. I’ll let it churn for a few days and we’ll see how helpful it is. Thanks for the app.

gentoo_biscuit,

This sounds awesome, I’ll give it a try! Would this work in i2p?

mgdigital,

I’ve never used I2P but I don’t see why not!

railsdev,

You had me GraphQL 🥰

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • selfhosted@lemmy.world
  • slotface
  • kavyap
  • thenastyranch
  • everett
  • tacticalgear
  • rosin
  • Durango
  • DreamBathrooms
  • mdbf
  • magazineikmin
  • InstantRegret
  • Youngstown
  • khanakhh
  • ethstaker
  • JUstTest
  • ngwrru68w68
  • cisconetworking
  • modclub
  • tester
  • osvaldo12
  • cubers
  • GTA5RPClips
  • normalnudes
  • Leos
  • provamag3
  • anitta
  • megavids
  • lostlight
  • All magazines
    •