In the previous series of articles on hot updates, the process and packaging details of UE hot updates were introduced. In fact, there are some engineering practices for optimizing hot update patches that I think can be discussed in detail.
This article starts with the generation of Shader patches, aiming to reduce the size of shaders during each hot update. It will analyze some internal implementation details of the engine, address issues related to Shader patches in the engine, and automate the Shader patch process based on HotPatcher.
2021.11.02 UPDATE: HotPatcher now supports collecting all compiled shaders of Cooked resources in the patch and packaging them into a Shader Library, which can replace the Shader Patch mechanism. For more details, see the article: UE Hot Updates: Shader Update Strategy.
Generation and Loading of Shader Bytecode
UE provides an option to Share Material Shader Code
in Project Settings
-Packaging
, which can control the sharing of shaders by storing them as separate files, thus reducing the package size.
By default shader code gets saved inline inside material assets, enabling this option will store only shader code once as individual files This will reduce overall package size but might increase loading time.
When this option is enabled and the project is packaged, two ushaderbytecode
files will be generated in the Content
directory of the package:
The Mount Point in the packaged Pak is:
1 | ../../../PROJECT_NAME/Content/ |
And the naming of the ushaderbytecode files follows the rules below:
1 | static FString ShaderExtension = TEXT(".ushaderbytecode"); |
The ushaderbytecode is automatically loaded when the engine starts, but note the difference in loading timing for Global and project:
1 | int32 FEngineLoop::PreInitPreStartupScreen(const TCHAR* CmdLine) |
In summary, the handling of shaders during engine packaging needs to pay attention to the following two points:
- The packaged project will only compile the shaders of the currently packaged resources into the ushaderbytecode.
- The engine will automatically load them on startup.
Therefore, if there are changes to shaders or new shaders are added during resource hot updates, failing to package the shaderbytecode will result in some resources not displaying correctly, as shown in the image:
Error in Log:
This requires us to handle updates to shaders.
Hot Updating Shaders
Based on the previous section’s introduction, we know that during packaging, the compiled shaders of the resources being packaged will be generated into the ushaderbytecode file. When we perform a hot update and add new materials, we need to ensure that the new shader files are included in the pak during runtime; otherwise, the material effects will be lost.
In HotPatcher, there is an option that includes shaderbytecode, packaging the latest generated ushaderbytecode after cooking into the pak:
And the Mount Point is the same as the base package.
When we mount the hot update pak, it needs to have a pak order greater than that of the basic package; thus, the ushaderbytecode in the hot updated pak will have the highest priority. However, as mentioned earlier, the engine automatically loads the shaderbytecode from the base package at startup, and the shaderbytecode from the mounted pak will not be automatically loaded after the program runs, requiring a manual load after mounting the pak:
1 | void UFlibPatchParserHelper::ReloadShaderbytecode() |
Simply call the FShaderCodeLibrary::OpenLibrary
function.
In summary, the workflow for hot updating shaders is as follows:
- Execute Cook to generate the latest resources’ ushaderbytecode file.
- Package the ushaderbytecode into the pak.
- Manually load the ushaderbytecode.
To regenerate the ushaderbytecode, you can use the following cook command:
1 | UE4Editor-cmd.exe PROJECT_NAME.uproject -run=cook -targetplatform=WindowsNoEditor -Iterate -UnVersioned -Compressed |
This will effectively rebuild metadata, and the AssetRegistry and similar files will be regenerated. After execution, the files Saved/Cooked
under AssetRegistry.bin
and the Metadata
directory /Content/ShaderArchive-*.ushaderbytecode
, as well as Ending/GlobalShaderCache*.bin
, will all be the latest generated ones, which can then be packaged using HotPatcher.
Generation of Shader Patches
However, the previous section’s introduction indicates that each hot update requires a complete update of the shaderbytecode, which can be wasteful because the shaders already exist in the base package. It is unnecessary to include them in every update, especially as the shader size can become quite large, with some reaching several hundred MB. Thus, we need to achieve incremental updates for shaders through patching.
Starting from version 4.23+, UE provides a method to create Shader Patches where Old Metadata and New Metadata directories must be provided. The Metadata
must have the following directory structure:
1 | D:\Unreal Projects\Blank425\Saved\Cooked\WindowsNoEditor\Blank425\Metadata>tree /a /f |
One must back up the Metadata directory at the time of packaging the base package, using the latest project’s Metadata directory post-Cook as New Metadata and the base package’s as Old Metadata. Then call the engine’s FShaderCodeLibrary::CreatePatchLibrary
function; however, this function has different prototypes in different engine versions, so a layer of encapsulation can be implemented:
1 | bool UFlibShaderPatchHelper::CreateShaderCodePatch(TArray<FString> const& OldMetaDataDirs, FString const& NewMetaDataDir, FString const& OutDir, bool bNativeFormat) |
The internal implementation of FShaderCodeLibrary::CreatePatchLibrary
compares the old Shader data serialized from Old Metadata with the New Metadata, taking the differing parts as the Shader in the Patch.
HotPatcher provides configuration and management methods for Shader Patches:
Multiple platforms and Metadata directory parameters can be specified simultaneously, and commandlet support is provided.
4.25+ Shader Patch Crash
Note: The FShaderCodeLibrary::CreatePatchLibrary
provided by the engine has a bug in 4.25 that can lead to a crash when generating Patches. Here’s a solution.
In the 4.25 engine version, calling FShaderCodeLibrary::CreatePatchLibrary
to create a Shader Code Patch will trigger a check exception:
This is because the constructor of FEditorShaderCodeArchive
calls ShaderHashTable’s Initialize and gives a default value of 0x1000
:
1 | FEditorShaderCodeArchive(FName InFormat) |
This leads to the subsequent process (FSerializedShaderArchive::Serialize
) calling Initialize
and triggering a failed check (because HashSize already has a value and is not 0). This check failure is due to the redundancy in initialization logic between FEditorShaderCodeArchive
and FSerializedShaderArchive::Serialize
, which doesn’t manage these two states correctly.
The resolution can be achieved by adding getter methods to retrieve the HashSize
and IndexSize
properties of FHashTable
:
1 | class FHashTable |
Then, detect in FSerializedShaderArchive::Serialize
if initialization has already occurred, and if so, skip the initialization logic.
Automated Shader Patch Process
HotPatcher supports configuration-based commandlet functionality for Shader Patches, which can be executed through a configuration file:
1 | UE4Editor.exe PROJECT.uproject -run=HotShaderPatch -config="export-shaderpatch-config.json" |
The -config
parameter accepts files that can be exported from the editor via plugins.
It also supports command line parameter replacement, similar to HotRelease and HotPatcher’s commandlet functionality. More details can be found in the HotPatcher documentation:
As long as you manage the Metadata directories for each version of your project, you can maintain a universal configuration file that can be used to generate Shader Patch files before the HotPatcher Patch tasks. The remaining task is to load the Shader Patch files during runtime, with the plugin providing function library support:
1 | bool UFlibPatchParserHelper::LoadShaderbytecode(const FString& LibraryName, const FString& LibraryDir); |
Notes
When using Shader Patches, it’s important to note that the generated Shader Patch’s ushaderbytecode
files will have the same filenames as those in the base package. Therefore, the default mounting during engine startup could lead to the ushaderbytecode files in the base package being undetectable, resulting in a crash.
You should handle loading the Shader Patch’s ushaderbytecode
files yourself after mounting, following the earlier instructions.
Additionally: Shader Patch updates do not directly support iterative patching, meaning you cannot generate a Shader Patch for 1.0 Metadata combined with a 1.1 Shader Patch to produce a 1.2 Shader Patch. You must rely on the complete Metadata from 1.1 to do so. Therefore, each patch must be based on the previous complete Metadata (both Project and Global ushaderbytecode files), requiring comprehensive Management of Metadata during each packaging process.