If you run "guix pull" today, you get a package graph of more than 22,000 nodes rooted in a 357-byte program---something that had never been achieved, to our knowledge, since the birth of Unix: a Full-Source Bootstrap.
@janneke not to underpaint the importance and coolness of this achievement, here's an uninformed question that you probably get a lot: how does this work wrt to depending on a Linux kernel (which is tons of C), some basic userland (or can it run as PID 1-and-only?), and x86 hardware (which... who knows what it does) to run this 357 byte binary?
If you can't trust a compiler to build your program correctly, why can you trust a kernel and some hardware to run your binary correctly?
There is currently no good answer to that other than that we chose to start on getting rid of the obviously unnecessary and "easy" binary seeds first. Or: different people have different interests and competences, if we start then eventually we'll probably get there someday. There are some ideas, though.
The least elegant but easiest "solution" would be to revert to Diverse Double Compliing (DDC, https://dwheeler.com/trusting-trust/). The low level tools (stage0, m2-planet, and mes) can easily do cross builds. You could build on different architectures, and kernels if you like and compare package checksums.
Running as PID 1: During the same RB-V conference, Ludovic Courtès prototyped building a Guix package in the initial ramdisk. After the build the package is discarded, but before that its checksum is printed and can be checked with a build under GNU/Linux.
Also, Stage0 was designed to also run on the Knight VM, one could imagine running that on simpler hardware, or running the VM on different machines/architectures, dunno.
I'm probably not Guix-savvy enough to fully comprehend the issue here — but as I understand it, you want to be super explicit about what each package needs to be built. Do you include libc, cc, binutils into this list of dependencies? (I imagine you do, otherwise it wouldn't be reproducible.) Apparently you do include /bin/sh.
So yeah, the Hurd servers aren't much different or any more "external" to the environment than /bin/sh. I don't think you should be firmlinking stuff from the host; you should probably just spawn a mini subhurd for each build. You want pipes and fork/exec, so you need pflocal, proc, and exec servers.
(Also /servers/proc, mentioned in your mail, is not a thing, of course 🙂 — the proc server is one of the two servers, the other one being auth, that are not accessible through the file system, but only through _hurd_ports.)
Your mail about /bin/sh also raises an interesting topic of paths. Do you want to change /dev/null and /servers/exec to some other (hash-derived I would imagine) paths? Sounds wild but you totally could!
You could then either patch glibc (and everyone who expects to find /dev/null at its usual place), or provide symlinks. But then again I don't know enough about Guix to judge here.
Unfortunately all this wouldn't help you too much with bootstrapping from source, since you cannot do I/O easily on the Hurd like you can on Linux with a few instructions; you need to do RPCs and all that (even to get your argv). This is of course hidden from you when you're using glibc.
@bugaevc Exactly! So the question becomes: assuming you have nothing but the Mach syscalls at your disposal, what chain of programs building on each other would eventually let you run a proc and an exec server so you have the beginning of a POSIX build environment?
The whole stage0/M2/Mes story on Linux was quite a puzzle; its Hurd version would push it further. :-)
> Also, one could argue that things like /dev/null have a well-defined interface that’s set in stone and that, consequently, how they’re implemented does not matter at all.
Yes, but also no: there certainly can be differences in behavior that are allowed by the interface (where it explicitly doesn't guarantee something), but (due to bugs) can influence the outcome. For instance, does every write to /dev/null always write the whole buffer, or can there be short writes?
@bugaevc The Hurd code lives in /gnu/store/…-hurd-, but the translation points in the build environment would remain /dev/ and /servers/*. Changing that would be impractical and bring nothing.
@janneke@bugaevc Actually I keep making the same mistake: there’s no /servers/proc but for some reason we have it in childhurds, just with no translator on it (I may be the guilty party :-)).
I was thinking of the following scheme, which I have not tried, so this is just a theory.
You create an executable (perhaps as an unnamed file) that is setuid to yourself, and then exec it (not over your own task, unless you want that), without passing an auth or proc ports (as you have none).
The translator notices this and creates a new auth handle based on its idea of your effective uids/gids (see libfshelp/exec-reauth.c); and then the exec server gives the new task a fresh proc port. You cannot access the new task because of setuid/EXEC_SECURE, but as you created the executable you still control what it does.
@bugaevc The build environment includes nothing bug the explicitly-declared userland dependencies. If a package depends on GCC and Binutils, it gets them; if not, it doesn’t.
There’s no /bin/sh there—no /bin, no /usr, nothing.
On Linux, there’s /dev and /proc, but for separate namespaces.
find an "open source hardware" board where you can somehow verify the hardware aren't playing games on you (in particular not running all of your code in a nearly undetectable hypervisor, like we know Intel does...), probably some RISC-V board
run you bootstrapping code on it with no OS whatsoever; hopefully it doesn't need much from the OS
you'd have to build in a serial driver or something like that (blinking LEDs is cool but you can't input program source this way), not that I have any idea about hardware
@theruran@janneke the Hurd surely can run GCC and cross-compile Linux; but I'm not sure you would be winning much, for two reasons:
It's nowhere near as trivial to do "syscalls" as on Linux — on Linux you place some values into some registers and perform "int 0x80" or "syscall", and that's it, you've called write or exit. On the Hurd, these all are implemented in glibc on top of Mach IPC, and that needs quite a lot of code to happen.
Linux is huge, but you can build it in a minimal configuration (see https://tiny.wiki.kernel.org/). Mach may be a microkernel, but it's minimal in functionality, not size. In fact it's a meme in the microkernel community just how large for a microkernel Mach is. But I don't have any numbers to quantify this.
@theruran@janneke another issue that arises once you take hardware into the picture is: you cannot cross-compile hardware; nor can you hash your hardware and compare it to some known-good hardware.
You may trust your board to really run the code you give it; you may manage to bootstrap a x86_64-linux-gnu GCC on it and confirm it's identical to what your distro ships. But that still doesn't guarantee you that your Intel processor actually runs what your binary says.
@bugaevc We need awareness without resignation: awareness that many other issues are yet to be addressed, while making progress in every way we can.
That Linux userland is “addressed” by what @janneke et al. have been doing is already a huge step forward, one long considered unachievable. Others will work on hardware and someday folks will meet halfway. :-)
For the hardware side of things, @bunnie has been making some awesome progress towards methods for trusting modern chips, for example the infra-red in situ method of silicon inspection: https://www.bunniestudios.com/blog/?p=6712
I tried to realize this before with my Kestrel Computer Project. But, zero people showed any willingness to help, because it was still kinda sorta too abstract. Or, maybe I was just too nostalgic (the idea was to "revert progress" and develop a neo-retro platform from which we could once again move forward from).
One of my core ideas was to standardize (memory-mapped) I/O register interfaces for a wide variety of peripheral classes using an I/O fabric like (or even forking) RapidIO, kind of like how the most sophisticated video cards of today still support true-blue VGA emulation. I reject the idea that it's "impossible" to achieve with other classes of peripherals.
But, that project is dead now due to (1) nobody willing to help out, and (2) me burning out.
@vertigo@bugaevc@theruran
Yeah. One of the things that helped a lot, especially in the first couple of years, was the enormous amount of mental support that I got.
@janneke@aziz@fsf@fsfe@reproducible_builds@ekaitz_zarraga Indeed! Right now we can bootstrap all the way from #hex0 to #Mes, then use #MesCC to build very first build of #tinycc (we can call it mes-tcc). mes-tcc can then build the next build of tinycc (boot0-tcc). Unfortunately, at the moment boot0-tcc segfaults. Today, I fixed one crash which was due to Global Offset Table being all zeros but it turns out we are now hitting another segfault, so more work is needed.
What I don't understand: such a bootstrap must've been done at least once before in history, otherwise we wouldn't have any compiled programs, right? Why wasn't it possible to go the route of the previous bootstrap again? I'd appreciate a link to some further reading on this. 🙏
@maltimore@fsf@fsfe
Some reasons for this include: The process wasn't documented, the code was lost, used many different hardwares, it took about 50 years, untangling history is hard.
For that last remark, just look at the Java or Rust bootstraps (they needed many, many steps) or the sheer impossibility to bootstrap the NPM/Node distaster.
@byterhymer@fsf@fsfe
For me personally (see the blog post) that would be: cleaning-up the FSB---we cut quite some corners---, getting rid of the ancient gcc-2.95.3 dependency (directly build gcc-4.6.4), getting Gash/Gash-Utils to run on top of Mes, and and RISC-V support, hopefully followed by ARM/AArch64.
But yeah, I really hope that others will address the hardware and kernel bits of the bootstrap!
@clacke
I'd like to think that I'm well funded which enables me to do things that are important and fun.
I'd also like to think that all my time is my own; technically my current Hurd work isn't funded, but that's how Mes started off too, so yeah.
@joeyh@janneke I had begun working on that part a few years ago: https://gitlab.com/giomasce/asmc. Then life happened and I ran out of steam for that, but hopefully I was not too far from compiling and running Linux at computer boot time, from a handful of KB of binary code.
@janneke holy shit I didn't know that this much progress had been made in bootstrapping. I thought we were still maybe years away from full source bootstrap. incredible work!
Add comment