@drewdevault Wait you implemented ext4? I assumed that for this kind of project you'd make a baby's first file system style thing like we did in OS class in uni...
@drewdevault Of course, just the idea of files having different entries in the FS table, but pointing to the same underlying data on disk is still something I find a little bit like black magic. 😂
I confess that I rushed into userspace without a plan, so I have a bit of a mess to figure out once I decide how to actually structure the project properly.
But that aside, I implemented a handful of syscalls today, notably readv, write, and mmap (MAP_ANON only, for allocations)
Okay, reorganized the source tree to make a lot more sense before we move any further into userspace. Took some inspiration from the BSD layouts, since I intend to ship kernel and userspace as one cohesive project.
Nice, that's the bulk of the scheduler refactoring done. No pretty screenshot since there are no user-visible changes, but the short of it is that we can context switch to and from the kernel rather than only to and from userspace.
(no, I have not set up kernel preemption yet, there are no locks in the kernel and interrupts are always disabled when the kernel is running; as a consequence of that I can only cooperatively context switch the kernel right now)
Elaborating a bit: the scheduler can now switch tasks while in the kernel and return to the kernel when the thread wakes up, and accordingly waitqueues have been implemented for this purpose.
The problem with console devices is that their file descriptors are fake, namely they do not store an inode reference. This is problematic for a few reasons, such as (1) Bunnix lacks a device file abstraction generally; (2) files without an inode blocks adding fstat; (3) we can't mount them in /dev.
The ext4 bug, by comparison, was annoying in that it took a long time to figure out, but in the end the fix was simple.
Whereas each time I have to work on the scheduler I have to redesign major portions of it and do heaps of rather difficult debugging of non-linear code.
@f4grx they're a better design, using O_DIRECTORY etc, imho. Particularly when it comes to implementing mountpoints. Simplifies things if you don't start from the assumption that you're always walking from the root
Ended up with some extra free time tonight and did a couple more syscalls: unlinkat, mkdirat
Remaining filesystem-related syscalls are the stat family of calls, rmdirat (or AT_REMOVEDIR I guess), chmod/chown/chgrp, readlinkat, linkat, symlinkat, and utimensat.
Implemented a bunch of openat flags this morning: RDONLY/WRONLY/RDWR is now enforced, and O_DIRECTORY, O_APPEND, O_TRUNC, and O_EXCL are now supported
I chose to diverge from POSIX on O_DIRECTORY because I think it's kind of lame, but I could change my mind if this comes back to bite me when I start porting real-world software
@drewdevault O_DIRECTORY even caused lots of discussion in the Filesystem working group of the POSIX standards committee.
Are file descriptiors obtained with O_DIRECTORY useful for sync_fd() in your implementation?
@mxk I haven't implemented fsync and presently all writes are committed to disk before write(2) et al returns to userspace, so I'm not sure. I might know more when I get to that.
@drewdevault@bugaevc okay, yeah, that is reasonable.
I ask, since I can't count the times I had to explain to someone, that O_RDONLY and O_DIRECTORY both are ways to create file descriptors to a directory, but both are fundamentally different in nature.
(And given that nobody checks return values of fsync() this mistake is unlikely to ever be discovered)
Also, reading from directory fds is a mess and shouldn't be supported by any posix system....
@drewdevault@mxk guess it can be useful — for atomicity, or for reducing the number of file name lookups in find(1)/ftw(3) if readdir returns DT_UNKNOWN — to be able to open() a node first, then fstat it, and if it's a directory do one thing (look up something relative to it with *at()), and do another thing otherwise.
The cache currently serves just to make sure that there is a 1:1 correspondence between inode numbers and inode data structures (so that if several files hold a reference to the same inode they all remain in sync), it does not persist inodes after all references are released... yet. A broader cache system that keeps cached data in unused RAM is planned for the future.
Simon Zeni ported the Helios EFI bootloader to the Bunnix boot protocol last night, so I'm doing some other boot improvements this morning to get EFI support upstreamed
New syscall: fstatat (currently AT_EMPTY_PATH only, i.e. fstat(2) behavior)
Serious build system improvements, to add...
A sysroot during the build, into which...
An early port of musl libc has been installed, including varying degrees of support for stdio, math, complex, ctype, malloc, string, unistd, stat, and more
Unfortunately I'm now fully committed to GNU make, but this build is too complex to really get away with POSIX makefiles.
Goal for the end of the week is to have a pretty decent C environment with a shell and a decent spread of coreutils ported, probably a bourne shell derivative plus sbase.
I'll port DOOM, too, if all of that is done before Saturday.
Progress towards shell: I have some dash patches that almost build; 11 libc functions remain. These are mostly going to involve shimming out signal-handling functions until I implement them in the kernel.
I have 44 coreutils from sbase building in my sysroot (e.g. grep, sed, od, uniq, etc)
A keyboard driver and console thing for an interactive shell remains to be done in the kernel
I fixed signals and the shell works great. So I went on a porting spree. I ported a further 15 core utilities from sbase, which includes ed(1), so now there is a text editor for Bunnix.
The real news, however, is that I somehow managed to port binutils, and you can assemble and link assembly programs on Bunnix now. I also got gcc to compile for Bunnix, but it has some bizzare issues I don't care to debug right now -- I didn't expect to get even that far.
I added system and popen/pclose added to libc so upstream ed works. Otherwise the only difference from upstream is that cron, cons, logger, nice, renice, time, and tftp were removed. Some stuff builds but doesn't work at runtime, like setsid.
@drewdevault what C standard does it target C89, C99, C11? Also if you don't mind answering how did you get GCC compiled on that system? Did you do a compiler bootstrap or are you able to just load the binary. One of my favorite parts of OSDev is watching people bootstrap a compiler, libc to start working on userspace
@drewdevault I've been keeping up with this thread a bit, I may have missed it but did you ever say what you were using for binaries? I'm curious now, are you doing https://en.wikipedia.org/wiki/A.out (not for you but for anyone reading) elf, or your own thing? ELF is very interestinghttps://kevinboone.me/elfdemo.html I love to see people writing programs in ELF. I wanted to ask you earlier about PIE and whether that was something you were planning on implementing. Not any pressure (i know it's just a side project) I just find operating system design very fascinating as a subject.
g++ also works, but adding libstdc++ to the initrd makes it too big and some stuff breaks. Could run it off of ext4 but there's some issues with the ext4 implementation (namely that binutils depends on lseeking beyond the end of the file to create a hole)
Just messing around, anyway. I don't intend to ship gcc in the base system (going with tcc instead). I do intend to ship binutils, though, so that you can have a working Hare toolchain, but that'll depend on fixing this lseek issue
Built out a small system for packaging ports in .tar.gz files and dumping them in /dist, plus a small script to automate their installation -- so I can ship some optional package sets.
First one is "devel", which includes make and tcc.
@drewdevault Do you really need EFI variables? You can just put your bootloader in esp://EFI/Boot/bootx64.efi. Which is what Asahi does due to a lack of EFI variables on their platform.
@drewdevault does your OS have a global notion of truth in terms of the Mount/Filesystem view, or is it alright connected to the individual process? (In preparation of chroot)
(Just to make getcwd more complicated)
@drewdevault Are there resources that you use or that you'd recommend to someone wanting to learn about these low-level OS topics? Perhaps some books / websites or something like that?
@drewdevault Sorry, no judgement here, just wanted to point out an interesting paper I'd read recently only for the first time. And that perhaps might influence how deeply one plumbs fork as an abstraction into the OS.
@drewdevault Dude you're amazing. I've been stuck on just memory management in my hobby kernel for like 6 months now (although truthfully like 6 hours I barely get to work on it), and here you are just pumping out a kernel like it's nothing.
@noodlez1232 I've done this a few times already, experience makes it easier :) and I still struggle with memory management! In a way it's the hardest part imho
Add comment