I'm curious to hear if anyone has successful strategies for debugging shader compile time issues (specifically #GLSL in #Vulkan in my case but still interested in hearing about others).
I've got this shader that takes over a minute to compile. There's various things I could do to prevent loop unrolling etc. but I still have to use trial and error to find the right place.
Do you have any better strategy than changing random places and seeing what makes a difference?
@aras@solidangle@beanz
Nice, thanks !
Well, this doesn't actually help for my specific issue which is glsl rather than hlsl, however definitely an interesting data point and giving me more insight.
I'm curious if there are ways of obtaining similar profiling data for Mesa's SPIR-V to NIR compiler.
@julienbarnoin@aras@solidangle@beanz You can just use regular profiling tool on Mesa to see which pass is taking a long time, there is also NIR_PRINT=1 (or something like that) to print out the NIR after each pass in a debug build to see if the code size blows up for some reason.
@crzwdjk@aras@solidangle@beanz@NOTimothyLottes
Thanks for the suggestions all !
So I haven't been able to profile the compiler yet due to life things, but from some trial and error checks, it's quite clear that I'm always plagued with the same issue: I need to stop passing large structs as inout function parameters in GLSL just to change a few members.
This is not a compiler-specific issue, this tends to take a long time to optimize for multiple ones in my experience.
@crzwdjk@aras@solidangle@beanz@NOTimothyLottes I'm not sure if this is a known GLSL trap, but I'm pretty sure I can build a simple pathological example that demonstrates it. I have a real-world code example that now takes over 10 minutes compiling.
After optimizing the result is alright I think, it all gets eliminated, but there's just so much code copying things around before optimizing that it takes forever to optimize out.
@crzwdjk@aras@solidangle@beanz@NOTimothyLottes
There's some related discussion here https://github.com/KhronosGroup/GLSL/issues/84, though the original complaint was from a performance perspective, I might submit a pathological example there that kills the Vulkan SDK's SPIR-V optimizer (and others) in terms of compile time to demonstrate why inout is just not an adequate replacement for pass-by-reference for some uses.
@julienbarnoin@aras@solidangle@beanz@NOTimothyLottes Ah yeah that would do it. Both GLSL and HLSL semantics say that inout function parameters are copied in at the beginning and out at the end. So yeah, passing it around as an inout is going to be messy, especially if the optimizer does things like break the structure into a bunch of scalars, then add code to copy each one. But I think SPIR-V does have the ability to pass by reference, and GLSL should expose it.
@crzwdjk@aras@solidangle@beanz@NOTimothyLottes It would be great if it did ! I know you mentioned making changes to GLSL a couple times, do you mind elaborating on your relationship with GLSL? Are you an external contributor or do you officially work on GLSL?
@beanz@julienbarnoin@aras@solidangle@NOTimothyLottes From the GLSL point of view, I kind of agree. Khronos is ultimately a vendor consortium and nobody there is too motivated to move the language itself forward, because vendors mostly want to show off their cool new features which can mostly be bolted onto the side. It has to come from users (really from engines) who aren't well represented there.
@crzwdjk@beanz@julienbarnoin@aras@solidangle@NOTimothyLottes I should also note that I do think MS needs to be more involved with Khronos and in a collaborative way. I just also think that HLSL is a mess of a language because it has become this messy amalgamation of disjointed ideas and concepts. We need a few years (or a decade) of polish and refinement to make it a sane cohesive language.
I welcome collaboration from outside MS to accomplish that goal.
@beanz@crzwdjk@julienbarnoin@aras@solidangle The important transition is that to having real pointers. Making HLSL on par with what is possible on the hardware in general purpose compute. Branch tables, function pointers, branch to address in register, etc.
@NOTimothyLottes@beanz@crzwdjk@julienbarnoin@aras@solidangle ignoring the technical debt in the compilers, data pointers aren’t terribly hard. Function pointers are hard to do efficiently because you can’t spill your full register file to a stack, and divergent function calls could be the biggest foot-gun in GPU programming.
Would be nice to support using a wave as a single thread of execution with an explicit lane mask which is only conditionally applied when required. No actual divergence. It makes everything a lot easier when wave ops can always use all lanes if necessary, and when it's always ok to readlane() on lane 0.
This doesn't exclude also building in a complex divergence setup, but have capacity for both.
Tongue in cheek, but you're giving me flashbacks to the explicit predication using "indicators" (a.k.a. flags) in the Calculation lines in RPG II. You could have up to three predicates apply to guard a line of code or output.
@julienbarnoin@aras@solidangle@beanz Are you getting killed in the IHV compiler(s) or in glslang? I've found that if I don't spirv-opt.exe for size, the IHV compiler chokes.
If I was to guess, you are going to need workarounds like translating loops from compile-time known loop counts to fake dynamic (load something the compiler doesn't know is dynamically static).
We've seen bad big-Oh algorithms in FXC and driver compilers.
spirv-opt algorithms were originally written for simplicity, but not obviously wasteful algorithms. If you see a quadratic or worse blowup, do file a bug.
The very first thing spirv-opt (or size or speed; almost the same recipes) is to inline everything; passing large structs around might be bad in any case.
@dneto@NOTimothyLottes@julienbarnoin@aras@solidangle@beanz this is effectively what DXC does too. I’ve got a post-it note on my monitor about trying a slightly different pass order in Clang to see if we can meaningfully simplify the code in functions before inlining.
DXC has an old LLVM 3.x bug where inlining can sometimes spend an insane amount of time propagating memory aliasing annotations. This can cause enormous compile time explosions (observed on the order of 10x).
@dneto@NOTimothyLottes@aras@solidangle@beanz
Still have to look at this closer and make an example I can share, but just from a quick test of copy-pasting of function calls in my shader and measuring the time spirv-opt takes, I see this, which seems a pretty tight match for quadratic time.
(This is not actually measuring the spirv-opt program but calling into the shaderc library with shaderc_optimization_level_performance, I assume this ends up being pretty much the same internally?)
Add comment