@Doomed_Daniel@mastodon.gamedev.place avatar

Doomed_Daniel

@Doomed_Daniel@mastodon.gamedev.place

Linux User, Programmer, Gamedev, Metalhead ♠️ AD(H?)D ♠️ he/him ♠️ formerly on Twitter and mastodon.technology with the same handle

This profile is from a federated server and may be incomplete. Browse more on the original instance.

steve, to random
@steve@discuss.systems avatar

I didn’t know there were international standard trolls, cool

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@steve isn't C++ an ISO standard?

rygorous, to random
@rygorous@mastodon.gamedev.place avatar

Nach fast 40 Jahren kann ich nicht weiter schweigen: der Autor von "Backe, backe Kuchen" ist ein Stümper und gemeiner Lügner, und die Welt muss es wissen!

Sieben Sachen zum Kuchen backen, sagst du? Sieben? Wirklich?

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@rygorous Endlich sagt's mal einer!

Doomed_Daniel, to random
@Doomed_Daniel@mastodon.gamedev.place avatar

Heretical question: Does it (still) make sense to use Git LFS instead of plain Git (possibly with shallow clones to avoid dragging in the whole history for all the big files)?

In my tests it seemed like that worked pretty ok?
They weren't super exhaustive, but for example one of the repos was about 29GB with LFS and had >450 commits just in the main branch.

Committing from shallow clones works (AFAIK didn't in the past) and after conversion repos are about half the size than with LFS

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad @dotstdy
life-hack on the server:
$ git ls-tree -r --object-only HEAD > /tmp/objectlist.txt
$ git pack-objects --progress --no-reuse-delta --no-reuse-object ./objects/pack/foo < /tmp/objectlist.txt

afterwards the blobless clone of current HEAD is relatively fast (2:35min instead of 7:50) because it can download the prepared pack; I assume/hope that after pushing more commits it'll still use that pack as base and send that + diffs or whatever (until next repack); could be done 1/week?

Doomed_Daniel, (edited )
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad setting pack.windowMemory=100m makes the blobless clone a lot faster (7:50min instead of ~20min), --depth=2 also faster (3:30min); core.compression doesn't matter

(also pinging @dotstdy - I think you were interested in git with big files w/o LFS?)

BTW, full clone is pretty fast (w/ GBit connection to server), I think because no repacking is needed, it just sends the pack as is.
IIRC the checkout-part in that case took a bit longer, about as long as unzipping the repos HEAD files?

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@ali1234 doesn't it have pack index files to know what object a commit is in?

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad @MouseByTheSea
Good point about the locking!

I'll check out the blogpost

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@micha self-hosted gitlab or gitlab.com? because at gitlab.com storage and traffic is ridiculously expensive ($60/month for a pack of 10GB storage and 20GB traffic)

We used Git for code and SVN for Assets in the past, currently everything in one repo (with LFS for assets), on self-hosted Gitlab, currently moving to self-hosted Forgejo (Gitea-fork).

But while converting an old SVN repo I was wondering if LFS really is needed..

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@ali1234
What do you mean with "advertise"? Is it not possible to somehow get the git history and then do shallow clone or fetch+checkout or sth with a specific commit?

Apart from that, if you're gonna have a big repo you'll end up hosting yourself anyway, because storage and traffic at the commercial git hosters is much more expensive than a VPS or even cheap dedicated server with lots of storage that you can run Gitea/Forgejo or similar on.

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad One problem is, similar with shallow clones, that at least the initial clone can take pretty long, because the server needs to create new objects - in my testcase, "remote: Compressing Objects" took about 15 minutes. I tried setting core.compression = 0 on the server but it didn't make a difference, so I think the real problem is not compressing them, but getting the contents out of the old objects (=> decompressing those?). Should try repacking on server with compression=0

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad I guess the main problem really is zlib/deflate - zstd would be much faster for both compressing and decompressing (at least if the compression level for compressing is is low enough)

And lz4 would be even faster of course (with worse compression, obviously), might still be a sane choice for those on-the-fly repacks when checking out though (still better than LFS being uncompressed?)

Doomed_Daniel, (edited )
@Doomed_Daniel@mastodon.gamedev.place avatar

.. probably because Git compresses its objects, while LFS seems to store the files it manages uncompressed - checking out might take a bit longer due to decompression, but that might be compensated by faster downloads due to smaller size.

Seems to me the only advantage of LFS is that the files can be stored on different servers than the rest of the Git repo (e.g. in S3-like object storage)?

UPDATE: Another advantage: LFS supports locking files, pure git doesn't (thanks @sinbad !)

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

Of course it's totally possible that I'm missing something, like that repos with huge files that get changed a lot get dog-slow after some time (though git repack might help then?) or something else, I've only toyed around with this for a day or so.

So if anyone knows of things that work well with LFS but not plain Git (and ideally how to reproduce them), I'd love to hear them so I can try it out :-)

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad Hmm I repacked with core.compression 0 and created a new repo on my local server (Atom C2558 based); it's indeed bigger (as one would expect with deflate level 0) - 9.7GB instead of 7.2GB as bare repo on server - but blobless clone isn't much faster (around 20min each, now it was ~20sec less).

What's really weird: A shallow clone (--depth=2) only takes about 5:42 minutes, even though the .git/ dir in shallow and blobless checkouts is about 5.4GB in both cases?!

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad @dotstdy
for comparison, same repo as LFS on same server: git clone takes 55s and doesn't stress the servers CPU too much.
However, repo on server is 423MB + 28GB LFS files, instead of 7.2-9.7 GB when not using LFS.

My current state is: It's prolly ok-ish not to use LFS, but you'll need a beefy server with enough RAM for shallow or blobless clones, and need to tweak git config on server (uploadpack.allowfilter=true ; uploadpack.allowAnySHA1InWant=true ; pack.windowMemory=100m)

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad @MouseByTheSea
blobless clone sounds pretty good, maybe even better: git clone --filter=blob:limit=1m or similar, to only omit "big" files (would be even better if one could filter by file ending or by "it's binary data"), so you still have the full history on code, but not binary data.
OTOH, apparently git blame still works (and downloads blobs on demand) so --filter=blob:none might be sufficient

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@ali1234 Good to know, I didn't even know it was possible to not support that!

distractal, to random
@distractal@mastodon.social avatar

Does anyone else find it weird that multiple venerable “town square” type social media web apps’ CEOs/owners seem to be speed running their app’s destruction the year before an election?

Just waiting for Discord at this point.

This really highlights how important the fediverse is.

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@distractal
I mean, Discord did this idiotic username change..

Doomed_Daniel, to random
@Doomed_Daniel@mastodon.gamedev.place avatar

I was wondering why git lfs migrate is so fscking slow on my machine (in either direction, i.e. both import and export).

Turns out, that was a regression in 3.0.2, which just happens to be the version that Ubuntu 22.04 LTS is shipping: https://github.com/git-lfs/git-lfs/issues/4750

Thankfully, packagecloud has an up-to-date version: https://packagecloud.io/github/git-lfs/install#manual-deb

After the update, `git lfs migrate export --everything --include="*.uasset ... etc" had more commits converted in 5min than 3.0.2 in 2h (when I cancelled it)

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

By the way, the whole migration (970 commits, ~30GB repo size before conversion) took 17:45 minutes now.

You can vote on ubuntu upgrading/patching the package at https://bugs.launchpad.net/ubuntu/+source/git-lfs/+bug/1973645 by logging into launchpad and clicking this:

sinbad, to random
@sinbad@mastodon.gamedev.place avatar

Going to have to jump start the car again, battery is completely dead even though I replaced it not that long ago.

I really should have fired it up more often in the last 6 months but we have both completely gone off the idea of driving anywhere now, we're both just biking all the time. I really must just sell it - I've been keeping it as a backup because "what if?", but even that's no use if it's dead every time we think to try it 🙄

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@sinbad hmm it might help to disconnect the battery when not using the car

dotstdy, to random
@dotstdy@mastodon.social avatar

I am now 0x20 oof

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@dotstdy Happy Birthday! :)

maisiesummers, to animals
@maisiesummers@mastodon.nz avatar



Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@maisiesummers
I fear if I ever meet a big cat in real life I'd probably get eaten because I'd let it sniff my hand and then try to scratch its head instead of playing dead or whatever one is supposed to do

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@maisiesummers
cats really are all the same, aren't they

karolherbst, to random
@karolherbst@chaos.social avatar

The most interesting part of Linus interaction with a bigot is, that those bigots "want to have the old Linus back", but they fail to see that it's "the old Linus":

Insulting people who should know better, just instead of kernel contributors and maintainers, it's bigots on the receiving end and they don't like it 🙃

Maybe they'll figure it out if they think about it for a second...

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@lazy_comet @karolherbst @ionizedgirl

see https://social.kernel.org/objects/60bcac97-e7c7-4899-a1b6-bb72196cddaf

Doomed_Daniel, to animals
@Doomed_Daniel@mastodon.gamedev.place avatar

Find yourself someone who loves you as much as Fellwurst loves the sun

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

Doomed_Daniel,
@Doomed_Daniel@mastodon.gamedev.place avatar

@fmhilton no, a leopard in disguise ;)

(in reality, she's a https://en.wikipedia.org/wiki/Bengal_cat and her fur looks extra awesome in the sun)

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • JUstTest
  • mdbf
  • everett
  • osvaldo12
  • magazineikmin
  • thenastyranch
  • rosin
  • tester
  • Youngstown
  • Durango
  • slotface
  • ngwrru68w68
  • kavyap
  • DreamBathrooms
  • megavids
  • InstantRegret
  • ethstaker
  • GTA5RPClips
  • tacticalgear
  • normalnudes
  • Leos
  • modclub
  • khanakhh
  • cubers
  • cisconetworking
  • anitta
  • provamag3
  • lostlight
  • All magazines
    •