Nvidia’s new app lets you precompile gaming shaders during machine idle time

Microsoft, Intel are also working on their own solutions for the issue.

See full article...

 

[bushrat011899]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

EDIT:
I think the wording above is implying I want games to distribute pre-compiled shaders to be installed with updates. That's not what I'm saying. I explicity mean once the game is updated, Steam/etc. itself can run each game's specific shader compilation tool. Steam already supports this kind of functionality through installScripts, they just don't have an option to have it run automatically during updates, and instead wait until the user hits play. Even that, I think, is vastly better as a user experience compared to pressing play, Steam saying the game is ready to play, having the game go full screen, and then being stuck at a compilation progress bar.

 

[Resistance]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

That would work on consoles, where there are only a few hardware and software variations that need to be accounted for. It doesn't work on PC where there are far more variations that need individually compiled shaders.

Edit: Unless by "install" you mean "compile"? In which case, shader compilation is a very resource intensive process, I don't miss the days when a random automatic update would slow my PC to a crawl while I'm using it. Such an update process would have to be carefully implemented to prevent issues.

 

[jonbob_newcastle]

Unfortunately, Nvidia warns that users will still have to generate shaders in-game after downloading a title for the first time. The Auto Shader Compiler system only generates the new shaders needed after subsequent driver updates following that first run of a new title.

So that’s why Atomic Heart is recompiling shaders nearly every time I run it. Driver updates.

It’s not a terrible idea but I’ll pass. Even if it only happens in idle time I’d rather not have my machine chug away at a task that might not be necessary. Why bother doing it if I don’t actually play the game again until four or five driver updates later? I’d just as soon live with the occasional delay and use it as an excuse to go and make a cuppa.

What will happen if my computer is in the middle of doing this when I sit down to some simple task? Will I have to wait while it finishes up? I’d rather have to wait a long time when I expect to than be delayed when I don’t.

 

[bushrat011899]

That would work on consoles, where there are only a few hardware and software variations that need to be accounted for. It doesn't work on PC where there are far more variations that need individually compiled shaders.

Edit: Unless by "install" you mean "compile"? In which case, shader compilation is a very resource intensive process, I don't miss the days when a random automatic update would slow my PC to a crawl while I'm using it. Such an update process would have to be carefully implemented to prevent issues.

Yes, that's why I specify quote around "install". The game already includes everything it needs to compile shaders, so split that off into a headless app that can be run by Steam/EGS/etc. Then. The user can have a nice tickbox in their settings for "run post-update tasks automatically". If the feature is on, shader compilation is a silent feature as intrusive as game updates already are. If it's off, at least now the store itself can provide a consistent UI for the experience. The problem right now is most games that compile shaders on launch will be full screen at their menu while doing it. At least if it was a headless app Steam/etc. runs, I can do other things uninterrupted in the meantime.

 

[Siosphere]

Honestly they need to revamp the entire shader compilation process. Rebuilding in the background is fine, but doesn't fix new games sitting at a shader compilation screen and then still suffering from shader compilation stutter.

There has to be a better and more efficient way than what we have been doing. The current way works great for platforms that can cache and distribute (consoles, steamdeck, etc), but it really does suck on PC.

 

[jonbob_newcastle]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

The shaders are different for different graphics cards - and TIL even driver versions - so including all the possible permutations in the base install is a non-starter.

I suppose the publisher could pre-compile a number of different versions and let players download the appropriate one but it’s just much simpler to have everyone do their own. It’s not something that needs to be done very often.

The problem right now is most games that compile shaders on launch will be full screen at their menu while doing it. At least if it was a headless app Steam/etc. runs, I can do other things uninterrupted in the meantime.

That would be handy. You can always alt-tab out of the game is doing its thing but being able to tell Steam or what have you to compile the shaders as part of the install process would be a boon.

 

[Resistance]

Yes, that's why I specify quote around "install". The game already includes everything it needs to compile shaders, so split that off into a headless app that can be run by Steam/EGS/etc. Then. The user can have a nice tickbox in their settings for "run post-update tasks automatically". If the feature is on, shader compilation is a silent feature as intrusive as game updates already are. If it's off, at least now the store itself can provide a consistent UI for the experience. The problem right now is most games that compile shaders on launch will be full screen at their menu while doing it. At least if it was a headless app Steam/etc. runs, I can do other things uninterrupted in the meantime.

Sorry, shader compilation would not be a silent feature as intrusive as game updates already are, even if you optimize it to run slower it still takes a ton of memory.

 

[bushrat011899]

The shaders are different for different graphics cards - and TIL even driver versions - so including all the possible permutations in the base install is a non-starter.

I suppose the publisher could pre-compile a number of different versions and let players download the appropriate one but it’s just much simpler to have everyone do their own. It’s not something that needs to be done very often.

Agreed, that's why I specify that post update installation would compile shaders, not install compiled shaders. I'm sure Steam could setup some kind of elaborate peer-to-peer distributed shader compilation and distribution system, but that's too hard for what I think is a pretty simple problem of just letting the game actually finish the installation process.

That would be handy. You can always alt-tab out of the game is doing its thing but being able to tell Steam or what have you to compile the shaders as part of the install process would be a boon.

Exactly, and this applies to any kind of post-update nonsense too. Some games strip away unused localization data, others might further decompress assets, whatever. I just find it frustrating how we went from software having an installer, whose entire job was to make the software ready to use, to store fronts which are just downloaders.

 

[MrSteve007]

As long as they don’t work to eliminate the need to reticulate splines before the game starts…

 

[YetAnotherPoster]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

The Steam Deck does exactly this and downloads shaders as just another update for each game. If it were to work on PCs then Steam would have to offer shaders for each GPU.

 

[bushrat011899]

Sorry, 63% is not the vast majority of users.

I don't know where you're getting that metric from but I'm totally fine with calling that enough to justify developing the option to automate shader compilation...

 

[Roonski]

For the vast majority of users, their computer is on overnight, and that's when Steam/Windows/etc. download and install updates.

Is that true? Maybe for gamers, but AFAIK most computers actually go to sleep overnight, and only wake up briefly to do updates if necessary. Certainly most laptops will work this way...

 

[Cat Killer]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

It's how it's worked on Linux since at least 2020. If you're unlucky, you download the shaders as an intermediate representation along with the game or game update, which then get compiled for your machine - in the background when the machine's idle, or when you launch the game. They get queued for recompilation on driver updates. If you're lucky, someone with similar hardware and driver version has already compiled them and uploaded them back to Valve, so you get the shaders already compiled without having to do anything. I don't know why Windows is so behind on this.

 

[norton_I]

Sorry, shader compilation would not be a silent feature as intrusive as game updates already are, even if you optimize it to run slower it still takes a ton of memory.

So pause the compilation while the computer is in active use? Delay sleep/hibernation temporarily to allow the compilation to complete if needed? It's not like there aren't tons of examples of potentially expensive background processes that we have learned to schedule effectively.

 

[bushrat011899]

Is that true? Maybe for gamers

Which is the only group of people routinely inconvenienced by shader compilation anyway.

 

[Resistance]

I don't know where you're getting that metric from but I'm totally fine with calling that enough to justify developing the option to automate shader compilation...

Look, we're pretty much on the same side the debate, there's no need to move goalposts. Automatic background shader compilation is a good idea and as I said in my very first comment on the matter:

Such an update process would have to be carefully implemented to prevent issues.

So why are you debating with me by saying there are ways to mitigate the inconvenience?

 

[FlyingGoat]

Is that true? Maybe for gamers, but AFAIK most computers actually go to sleep overnight, and only wake up briefly to do updates if necessary. Certainly most laptops will work this way...

If your PC is automatically going into suspend mode (as opposed to as a result of user input), I don't believe it's that difficult for an app to delay entering suspend mode, and do the work then. Or, heck, could even notice when the PC is idle-ish, and do work in the background at low priority, pausing when no longer idle, though that's of course a fair bit more difficult.

 

[RickRoyLeonPrisZhoraRacha]

Not related to shaders, but with nVidia drivers, I tend to manually get them and usually stay one version behind. Reading the article, I then visited the nVidia driver site, and got the latest (595.97 for Windows 11) to install. Still had to smirk that Microsoft will flag the install as not from the MSFT Store and to approve installation. And nVidia is going to work with Microsoft on Advanced Shader Delivery? They need to get on page with Microsoft flagging their installer.

 

[reyan]

Yes, that's why I specify quote around "install". The game already includes everything it needs to compile shaders, so split that off into a headless app that can be run by Steam/EGS/etc. Then. The user can have a nice tickbox in their settings for "run post-update tasks automatically". If the feature is on, shader compilation is a silent feature as intrusive as game updates already are. If it's off, at least now the store itself can provide a consistent UI for the experience. The problem right now is most games that compile shaders on launch will be full screen at their menu while doing it. At least if it was a headless app Steam/etc. runs, I can do other things uninterrupted in the meantime.

So are you saying steam would monitor for driver updates? It isn’t only game updates that trigger this.

 

[bushrat011899]

So are you saying steam would monitor for driver updates? It isn’t only game updates that trigger this.

I mean Steam Hardware Survey already does monitor your driver version, wouldn't be hard to check it once a night at 2am and, if so, run shader compilation for relevant games. Regardless of whose update breaks the cache, this is a problem that I think should be solved when the update is installed, not deferred to when the user is actually trying to use the software in question.

 

[SeanJW]

Surely the real solution to this problem is to just "install" the shaders during game updates? We've already solved the timing problem for downloading the update itself, so just have the software distribution platform trigger a post-update installation that compiles shaders.

EDIT:
I think the wording above is implying I want games to distribute pre-compiled shaders to be installed with updates. That's not what I'm saying. I explicity mean once the game is updated, Steam/etc. itself can run each game's specific shader compilation tool. Steam already supports this kind of functionality through installScripts, they just don't have an option to have it run automatically during updates, and instead wait until the user hits play. Even that, I think, is vastly better as a user experience compared to pressing play, Steam saying the game is ready to play, having the game go full screen, and then being stuck at a compilation progress bar.

I think the problem is there's two points in time where shaders need to be compiled - both the app installation/update time (which is great, that'd be the perfect time after all), and when drivers get installed/updated. During the first you can cleanly just tack the "let's run a compile after the install" on the end; pretty much every automated install system has post-install script running somewhere. That can be solved by developers no problem. The latter has to be handled by a whole new process - the driver needs to know who and how to call the compile process during it's post-install phase. The ability to predownload which they're all adding on is just icing on the cake to speed things up.

Because there's no latter part right now, and everyone is scrambling to do something about it, we're stuck with the state of the art being the game going "shit... did the driver change? I have to recompile the shaders now" at the application start or on the fly during game execution, which will hopefully be a smaller but more frequent job that handles at suitable checkpoints where you're loading assets anyway and so a delay is to be expected.

Edit: The good part about the new systems is currently games don't have a clue whether a driver update actually mandates a recompile. So if the driver version changes, they probably punt to the worst case and off they go. Whereas if the driver is in control of when they recompile, hopefully silly point releases that have no impact will skip the whole deal.

 

[SeanJW]

So pause the compilation while the computer is in active use? Delay sleep/hibernation temporarily to allow the compilation to complete if needed? It's not like there aren't tons of examples of potentially expensive background processes that we have learned to schedule effectively.

hahahahaha... sorry, you lost me with that last sentence. We schedule them terribly. A truly grotesque number of them all run immediately on system start or resume, which is the worst time to run things, because presumably you started the system to actually do something rather than run all the other crap. It's a typical process to check for updates during application start up after all and that's just a hinderance to the user, though there's a thought process that makes it seem sensible. There's certainly good reasons for it (things may not work without the latest update, there might be security problems etc...) but it being something that happens always is utterly appalling.

(I don't have answers. I think the check should still take place, and it should return a "minimum version you must run" and "latest version you should run", and if the minimum is met, delay the update to the latest some interval after startup that's randomised a bit so you don't interfere with the user and there's no thundering herd of updates, and the user can still force things if you're notifying them or if there's compliance requirements it can be setting... but that's just some musings on my part. I don't know what other issues would come up, and need solutions)

Edit: Oh, and I'm sure there'd still be shitty developers who make the two things identical because it's lazy and solves their problems while offloading it to end users. But it's not like shitty developers are a new thing.

Edit 2: Sorry, you're not to blame, you just triggered one of my pet peeves. I switch laptops a lot, and nothing is more frustrating than going "fuck, I need to use my Windows/Mac laptop for XXXX specific task", picking it up then having to wait 30 minutes while it goes completely crazy catching up on updates.

 

[odikweos]

Honestly they need to revamp the entire shader compilation process. Rebuilding in the background is fine, but doesn't fix new games sitting at a shader compilation screen and then still suffering from shader compilation stutter.

There has to be a better and more efficient way than what we have been doing. The current way works great for platforms that can cache and distribute (consoles, steamdeck, etc), but it really does suck on PC.

It seems to me that they need to compile one reference shader per (game, hardware, driver, etc) tuple, and hash the resulting asm, and then go grab the rest from an online cache. Then pretty much only one person has to suffer the pain per unique config, and there are only so many unique configs. Hard to understand why that still hasn't been implemented in this day and age.

 

[SeanJW]

Is that true? Maybe for gamers, but AFAIK most computers actually go to sleep overnight, and only wake up briefly to do updates if necessary. Certainly most laptops will work this way...

That can happen if the conditions are right. Because it was never baked into Windows or PC hardware (well considering Windows predates common internet and update capability...), there's a hell of a lot of conservative design around it, so it doesn't happen as much as you'd think. And even then there's horror stories about laptops cooking in backpacks because they wake up to do the job and then can't go back to sleep because of some stupid application...

 

[+Griz]

Edit 2: Sorry, you're not to blame, you just triggered one of my pet peeves. I switch laptops a lot, and nothing is more frustrating than going "fuck, I need to use my Windows/Mac laptop for XXXX specific task", picking it up then having to wait 30 minutes while it goes completely crazy catching up on updates.

memories of living in a share house with 2mbps rural DSL and all the times we wanted to watch something on netflix but nope it's time for SYSTEM UPDATE and we couldn't even use another device instead because the console was using all the bandwidth

I haven't even updated my nvidia drivers since last January because of all the reports of bad drivers since then that cause crashing or random nonsense like all the fans running at 100% all the time. my card is 5 years old so it's not like I'm missing out on any of the new features

 

[SeanJW]

It's how it's worked on Linux since at least 2020. If you're unlucky, you download the shaders as an intermediate representation along with the game or game update, which then get compiled for your machine - in the background when the machine's idle, or when you launch the game. They get queued for recompilation on driver updates. If you're lucky, someone with similar hardware and driver version has already compiled them and uploaded them back to Valve, so you get the shaders already compiled without having to do anything. I don't know why Windows is so behind on this.

Most Linux package systems have the ability to link post-install scripts for other packages to a package update (eg. you can reprocess the man page caches after a package updates itself), but that's the sort of thing that's being talked about here for Windows specifically for driver updates and games. Linux just uses a general case that's already in place. Windows packaging is a total shit-show because there's a million different ways to do things.

Edit: My Linux laptop it's mostly apt doing the job. There's one program on it that decides to do it's own thing and self update out of ~/bin/ but I don't care enough to try and repackage it myself.

My Windows laptop, it's Windows Update, Microsoft Store, winget upgrade -all, and several applications that like to self update in their own happy way.

My Mac Laptop it's a system update, an app store update, and then dozens of the bloody things that use sparkle and go their own way....


Edit 2: Oh, and on Windows and Mac there's one application I have to manually check, and then unzip to the right place, but that's partially my fault seeing it's my application. On Linux it's easy to package and stuff in repos, but Windows and Mac are more tricky, especially with signing requirements so I just punt the problem seeing it only affects me.

 

[SplatMan_DK]

It seems to me that they need to compile one reference shader per (game, hardware, driver, etc) tuple, and hash the resulting asm, and then go grab the rest from an online cache. Then pretty much only one person has to suffer the pain per unique config, and there are only so many unique configs. Hard to understand why that still hasn't been implemented in this day and age.

Easy answer:

Nobody with a significant economic interest has identified a way to profit from it. So they just can't be arsed.

Current method makes each consumer pay for the process, with their time, power, GPU usage. Explain to megacorps CEO why they should change that (use the language of a child aged 8-10).

 

[odikweos]

Easy answer:

View attachment 132036

Nobody with a significant economic interest has identified a way to profit from it. So they just can't be arsed.

Current method makes each consumer pay for the process, with their time, power, GPU usage. Explain to megacorps CEO why they should change that (use the language of a child aged 8-10).

Sorry I was unclear. I didn't mean the company spending compute time precomputing shaders. I just mean after your box finishes, it checks the first shader hash against something other people uploaded, and if your hash is already there, you get the other ones. If you are the first one, you upload when you have a full set (for that game). That's it. NVIDIA has plenty of money for a little cloud storage, hence the constant driver updates. They could push them out every six months like it was back in the day, they upped the delivery rate for no real benefit to them beyond PR.

 

[Octavus]

Who is ch

It seems to me that they need to compile one reference shader per (game, hardware, driver, etc) tuple, and hash the resulting asm, and then go grab the rest from an online cache. Then pretty much only one person has to suffer the pain per unique config, and there are only so many unique configs. Hard to understand why that still hasn't been implemented in this day and age.

Who is checking that the one person who compiles and uploads the compiled shaders doesn't upload malware that runs on the GPU?

 

[stefan_lec]

As long as they don’t work to eliminate the need to reticulate splines before the game starts…

Hah! Came here to say the exact same thing. Have an upvote good sir!!

 

[LordEOD]

I only recently learned that the default global setting in nvidia control panel for shader cache was 1GB. Having to write (and overwrite) data to that cache, especially during game play could cause some bad shutters and dropped frames.

I only came across it whilst looking for remedies from horrible micro-stutter and hang-ups during my VR flying in MS flight simulator and (to a lesser degree) DCS. (Valve Index)
There were a few tweaks, but increasing the shader cache (in my case, to 10GB) made a big difference and the game has worked smoothly since.

To be fair, I never experienced that in any other title, hence why I never even looked at that setting before - still, feels to me that a 1GB default size is maybe too conservative given today's many unoptimized titles, but I digress.

Edit: Spelling

 

[Cold Fussion]

Honestly they need to revamp the entire shader compilation process. Rebuilding in the background is fine, but doesn't fix new games sitting at a shader compilation screen and then still suffering from shader compilation stutter.

There has to be a better and more efficient way than what we have been doing. The current way works great for platforms that can cache and distribute (consoles, steamdeck, etc), but it really does suck on PC.

The problem is most acute on unreal engine and with developers who develop a million shader permutations and then relying on Epic's tooling to capture those shader permutations. Epic needs better tools to capture the shader permutations and possibly unreal developers need to be mindful of the number of shader permutations. The problem is not insurmountable because other engines and studios have sold this problem long ago.

 

[kickahaota]

I'm glad that NVIDIA is putting an emphasis on this problem.

The real issue isn't that shaders need to be recompiled after an update. Okay, that's annoying, but it's a tiny cost per shader. The real issue is that, frequently, that shader isn't recompiled until the moment that it's needed in-game. The shaders for a monster get recompiled the first time that monster variety pops into view, and so on. This causes "micro-stutter", a hitch in the game's frame rate. That's a big problem. There are gamers who care about high frame rates and gamers who don't; but even the "30 frames per second is fine" gamers frequently don't like variation in frame rates. They especially don't like variations in frame rates exactly at the moment that a monster is about to eat their face.

These days, quite a number of games try to get ahead of this problem, by checking for out-of-date shaders as soon as the game loads up. But according to some developer interviews I've read, even if a developer is actively trying to do this, it's amazingly easy to get it work. Shaders are intentionally designed to be something that don't have to be micromanaged at runtime; you attach a shader to a graphical object during development, and let the drivers (and the game engine -- Unity, Unreal, etc.) handle automatically loading the shader when it's needed. Game code doesn't normally need to say "Wait, let me look at all the shaders at once here." And, of course, if it turns out that there's a bug in your shader-checking code, the game isn't going to crash, or fail to load the object, or do anything else that would leap out at you during QA testing. It's just that some of your objects are going to sometimes be a bit slow to load, and your frame rate will be just a bit hitchy, and gamers will be just a bit cranky.

If I'm right about this, then another useful thing for NVIDIA to do would be to make it a lot easier to spot the problem. Let the run-time game code set a flag that says "Okay, I think all the shaders are up-to-date"... and if a shader needs to be recompiled while that flag is set, let the driver send a telemetry event that says "...nope, you were wrong, here's the ID of the shader that needed recompiling and the ID of the graphical object that used it."

 

[Causification]

I don't understand why this has suddenly become a problem. For the over a decade I was playing PC games from hard drives, game startup times were around a minute. When I got my first SSD, startup times dropped to less than ten seconds. They stayed there for over a decade. Now suddenly every new game starts up like an octogenarian reorganizing their sock drawer. Halo Infinite, loading from a very nice NVMe SSD, takes over ten god damned minutes to get into gameplay, the vast majority of which is staring at a splash screen while the world's widest loading bar slowly fills up. What in the blue blazes of fuck are developers doing!?

 

[randomuser42]

I don't understand why this has suddenly become a problem. For the over a decade I was playing PC games from hard drives, game startup times were around a minute. When I got my first SSD, startup times dropped to less than ten seconds. They stayed there for over a decade. Now suddenly every new game starts up like an octogenarian reorganizing their sock drawer. Halo Infinite, loading from a very nice NVMe SSD, takes over ten god damned minutes to get into gameplay, the vast majority of which is staring at a splash screen while the world's widest loading bar slowly fills up. What in the blue blazes of fuck are developers doing!?

I have a question related to this. A couple of weeks ago I was playing BG3 on Windows. And I've been playing it for a while, through various driver updates, and I've never noticed this long compiling shaders step before.

I recently installed Linux (Cachyos if anyone cares!) and was a bit surprised to see it do that shader compiling step (and it's already happened at least once more after a driver update). I wrote it off as a quirk of Linux, but now I read this and apparently it's common?

I did see tips for how to bypass it on Linux and have it done on the fly though, and now to your post, it only happens on driver updates and it takes like maybe 2 minutes so it's not a big deal. I played Halo Infinite (on Windows) and don't remember any truly excessively long loading times. I wouldn't play a game with routine 10 minute load times, I'd assume that's something not working correctly.

 

[OptimusP83]

their computer is on overnight

Lolwut? How do you even begin to make that assertion...