Z's Blog

UE热更新：Create Shader Patch

Posted on In , Views:
Symbols count in article: 7.1k Reading time ≈ 18 mins.

In the previous series of articles on hot updates, the process and packaging details of UE hot updates were introduced. In fact, there are some engineering practices for optimizing hot update patches that I think can be discussed in detail.

This article starts with the generation of Shader patches, aiming to reduce the size of shaders during each hot update. It will analyze some internal implementation details of the engine, address issues related to Shader patches in the engine, and automate the Shader patch process based on HotPatcher.

2021.11.02 UPDATE: HotPatcher now supports collecting all compiled shaders of Cooked resources in the patch and packaging them into a Shader Library, which can replace the Shader Patch mechanism. For more details, see the article: UE Hot Updates: Shader Update Strategy.

Generation and Loading of Shader Bytecode

UE provides an option to Share Material Shader Code in Project Settings-Packaging, which can control the sharing of shaders by storing them as separate files, thus reducing the package size.

By default shader code gets saved inline inside material assets, enabling this option will store only shader code once as individual files This will reduce overall package size but might increase loading time.

When this option is enabled and the project is packaged, two ushaderbytecode files will be generated in the Content directory of the package:

The Mount Point in the packaged Pak is:

1
../../../PROJECT_NAME/Content/

And the naming of the ushaderbytecode files follows the rules below:

Runtime/RenderCore/Private/ShaderCodeLibrary.cpp
1
2
3
4
5
6
7
8
static FString ShaderExtension = TEXT(".ushaderbytecode");
static FString StableExtension = TEXT(".scl.csv");
static FString PipelineExtension = TEXT(".ushaderpipelines");

static FString GetCodeArchiveFilename(const FString& BaseDir, const FString& LibraryName, FName Platform)
{
	return BaseDir / FString::Printf(TEXT("ShaderArchive-%s-"), *LibraryName) + Platform.ToString() + ShaderExtension;
}

The ushaderbytecode is automatically loaded when the engine starts, but note the difference in loading timing for Global and project:

Runtime/Launch/Private/LaunchEngineLoop.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
int32 FEngineLoop::PreInitPreStartupScreen(const TCHAR* CmdLine)
{
  // ...
  {
    bool bUseCodeLibrary = FPlatformProperties::RequiresCookedData() || GAllowCookedDataInEditorBuilds;
    if (bUseCodeLibrary)
    {
      {
        SCOPED_BOOT_TIMING("FShaderCodeLibrary::InitForRuntime");
        // Will open material shader code storage if project was packaged with it
        // This only opens the Global shader library, which is always in the content dir.
        FShaderCodeLibrary::InitForRuntime(GMaxRHIShaderPlatform); // Load Global ushaderbytecode
      }

  #if !UE_EDITOR
      // Cooked data only - but also requires the code library - game only
      if (FPlatformProperties::RequiresCookedData())
      {
        SCOPED_BOOT_TIMING("FShaderPipelineCache::Initialize");
        // Initialize the pipeline cache system. Opening is deferred until the manual call to
        // OpenPipelineFileCache below, after content pak's ShaderCodeLibraries are loaded.
        FShaderPipelineCache::Initialize(GMaxRHIShaderPlatform);
      }
  #endif //!UE_EDITOR
    }
  }
  // ...
}

int32 FEngineLoop::PreInitPostStartupScreen(const TCHAR* CmdLine)
{
  // ...
  //Handle opening shader library after our EarlyLoadScreen
  {
    LLM_SCOPE(ELLMTag::Shaders);
    SCOPED_BOOT_TIMING("FShaderCodeLibrary::OpenLibrary");

    // Open the game library which contains the material shaders.
    FShaderCodeLibrary::OpenLibrary(FApp::GetProjectName(), FPaths::ProjectContentDir()); // Load project ushaderbytecode
    for (const FString& RootDir : FPlatformMisc::GetAdditionalRootDirectories())
    {
      FShaderCodeLibrary::OpenLibrary(FApp::GetProjectName(), FPaths::Combine(RootDir, FApp::GetProjectName(), TEXT("Content")));
    }

    // Now our shader code main library is opened, kick off the precompile, if already initialized
    FShaderPipelineCache::OpenPipelineFileCache(GMaxRHIShaderPlatform);
  }
  // ...
}

In summary, the handling of shaders during engine packaging needs to pay attention to the following two points:

  1. The packaged project will only compile the shaders of the currently packaged resources into the ushaderbytecode.
  2. The engine will automatically load them on startup.

Therefore, if there are changes to shaders or new shaders are added during resource hot updates, failing to package the shaderbytecode will result in some resources not displaying correctly, as shown in the image:

Error in Log:

This requires us to handle updates to shaders.

Hot Updating Shaders

Based on the previous section’s introduction, we know that during packaging, the compiled shaders of the resources being packaged will be generated into the ushaderbytecode file. When we perform a hot update and add new materials, we need to ensure that the new shader files are included in the pak during runtime; otherwise, the material effects will be lost.

In HotPatcher, there is an option that includes shaderbytecode, packaging the latest generated ushaderbytecode after cooking into the pak:

And the Mount Point is the same as the base package.

When we mount the hot update pak, it needs to have a pak order greater than that of the basic package; thus, the ushaderbytecode in the hot updated pak will have the highest priority. However, as mentioned earlier, the engine automatically loads the shaderbytecode from the base package at startup, and the shaderbytecode from the mounted pak will not be automatically loaded after the program runs, requiring a manual load after mounting the pak:

1
2
3
4
5
void UFlibPatchParserHelper::ReloadShaderbytecode()
{
	FShaderCodeLibrary::OpenLibrary("Global", FPaths::ProjectContentDir());
	FShaderCodeLibrary::OpenLibrary(FApp::GetProjectName(), FPaths::ProjectContentDir());
}

Simply call the FShaderCodeLibrary::OpenLibrary function.

In summary, the workflow for hot updating shaders is as follows:

  1. Execute Cook to generate the latest resources’ ushaderbytecode file.
  2. Package the ushaderbytecode into the pak.
  3. Manually load the ushaderbytecode.

To regenerate the ushaderbytecode, you can use the following cook command:

1
UE4Editor-cmd.exe PROJECT_NAME.uproject -run=cook -targetplatform=WindowsNoEditor -Iterate -UnVersioned -Compressed

This will effectively rebuild metadata, and the AssetRegistry and similar files will be regenerated. After execution, the files Saved/Cooked under AssetRegistry.bin and the Metadata directory /Content/ShaderArchive-*.ushaderbytecode, as well as Ending/GlobalShaderCache*.bin, will all be the latest generated ones, which can then be packaged using HotPatcher.

Generation of Shader Patches

However, the previous section’s introduction indicates that each hot update requires a complete update of the shaderbytecode, which can be wasteful because the shaders already exist in the base package. It is unnecessary to include them in every update, especially as the shader size can become quite large, with some reaching several hundred MB. Thus, we need to achieve incremental updates for shaders through patching.

Starting from version 4.23+, UE provides a method to create Shader Patches where Old Metadata and New Metadata directories must be provided. The Metadata must have the following directory structure:

1
2
3
4
5
6
7
8
9
10
11
D:\Unreal Projects\Blank425\Saved\Cooked\WindowsNoEditor\Blank425\Metadata>tree /a /f
Volume in drive C has no label.
Volume Serial Number is 0C49-9EA3
C:.
|   BulkDataInfo.ubulkmanifest
|   CookedIniVersion.txt
|   DevelopmentAssetRegistry.bin
|
\---ShaderLibrarySource
        ShaderArchive-Global-PCD3D_SM5.ushaderbytecode
        ShaderArchive-Blank425-PCD3D_SM5.ushaderbytecode

One must back up the Metadata directory at the time of packaging the base package, using the latest project’s Metadata directory post-Cook as New Metadata and the base package’s as Old Metadata. Then call the engine’s FShaderCodeLibrary::CreatePatchLibrary function; however, this function has different prototypes in different engine versions, so a layer of encapsulation can be implemented:

1
2
3
4
5
6
7
8
9
10
11
12
bool UFlibShaderPatchHelper::CreateShaderCodePatch(TArray<FString> const& OldMetaDataDirs, FString const& NewMetaDataDir, FString const& OutDir, bool bNativeFormat)
{
#if ENGINE_MINOR_VERSION > 25
	return FShaderCodeLibrary::CreatePatchLibrary(OldMetaDataDirs,NewMetaDataDir,OutDir,bNativeFormat,true);
#else
	#if ENGINE_MINOR_VERSION > 23
		return FShaderCodeLibrary::CreatePatchLibrary(OldMetaDataDirs,NewMetaDataDir,OutDir,bNativeFormat);
	#else
		return false;
	#endif
#endif
}

The internal implementation of FShaderCodeLibrary::CreatePatchLibrary compares the old Shader data serialized from Old Metadata with the New Metadata, taking the differing parts as the Shader in the Patch.

HotPatcher provides configuration and management methods for Shader Patches:

Multiple platforms and Metadata directory parameters can be specified simultaneously, and commandlet support is provided.

4.25+ Shader Patch Crash

Note: The FShaderCodeLibrary::CreatePatchLibrary provided by the engine has a bug in 4.25 that can lead to a crash when generating Patches. Here’s a solution.

In the 4.25 engine version, calling FShaderCodeLibrary::CreatePatchLibrary to create a Shader Code Patch will trigger a check exception:

This is because the constructor of FEditorShaderCodeArchive calls ShaderHashTable’s Initialize and gives a default value of 0x1000:

1
2
3
4
5
6
7
8
9
10
FEditorShaderCodeArchive(FName InFormat)
    : FormatName(InFormat)
    , Format(nullptr)
{
  Format = GetTargetPlatformManagerRef().FindShaderFormat(InFormat);
  check(Format);

  SerializedShaders.ShaderHashTable.Initialize(0x10000);
  SerializedShaders.ShaderMapHashTable.Initialize(0x10000);
}

This leads to the subsequent process (FSerializedShaderArchive::Serialize) calling Initialize and triggering a failed check (because HashSize already has a value and is not 0). This check failure is due to the redundancy in initialization logic between FEditorShaderCodeArchive and FSerializedShaderArchive::Serialize, which doesn’t manage these two states correctly.

The resolution can be achieved by adding getter methods to retrieve the HashSize and IndexSize properties of FHashTable:

1
2
3
4
5
6
7
8
class FHashTable
{
public:
  // ...
  FORCEINLINE uint32 GetHashSize() const { return HashSize; }
  FORCEINLINE uint32 GetIndexSize() const { return IndexSize; }
  // ...
};

Then, detect in FSerializedShaderArchive::Serialize if initialization has already occurred, and if so, skip the initialization logic.

Automated Shader Patch Process

HotPatcher supports configuration-based commandlet functionality for Shader Patches, which can be executed through a configuration file:

1
UE4Editor.exe PROJECT.uproject -run=HotShaderPatch -config="export-shaderpatch-config.json"

The -config parameter accepts files that can be exported from the editor via plugins.

It also supports command line parameter replacement, similar to HotRelease and HotPatcher’s commandlet functionality. More details can be found in the HotPatcher documentation:

As long as you manage the Metadata directories for each version of your project, you can maintain a universal configuration file that can be used to generate Shader Patch files before the HotPatcher Patch tasks. The remaining task is to load the Shader Patch files during runtime, with the plugin providing function library support:

1
bool UFlibPatchParserHelper::LoadShaderbytecode(const FString& LibraryName, const FString& LibraryDir);

Notes

When using Shader Patches, it’s important to note that the generated Shader Patch’s ushaderbytecode files will have the same filenames as those in the base package. Therefore, the default mounting during engine startup could lead to the ushaderbytecode files in the base package being undetectable, resulting in a crash.

You should handle loading the Shader Patch’s ushaderbytecode files yourself after mounting, following the earlier instructions.

Additionally: Shader Patch updates do not directly support iterative patching, meaning you cannot generate a Shader Patch for 1.0 Metadata combined with a 1.1 Shader Patch to produce a 1.2 Shader Patch. You must rely on the complete Metadata from 1.1 to do so. Therefore, each patch must be based on the previous complete Metadata (both Project and Global ushaderbytecode files), requiring comprehensive Management of Metadata during each packaging process.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:UE热更新：Create Shader Patch
Author:LIPENGZHA
Publish Date:2021/03/12 09:49
World Count:7.1k Words
Link:https://en.imzlp.com/posts/5867/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!