Z's Blog

In the article ZSTD Dictionary-based Shader Compression Scheme, I introduced a method for compressing UE’s ShaderCode using ZSTD dictionaries, which can significantly improve the compression ratio of ShaderLibrary. However, the process of training the dictionary and using it for compression remains complex and is not an efficient engineering implementation:

1. The engine’s default Shader compression must be turned off.
2. You need to cook the project once to dump the ShaderCode of each unique Shader.
3. Based on the dumped ShaderCode files, use the ZSTD program to train the dictionary.
4. Execute a full cook process again to compress using the dictionary and generate the final shaderbytecode.

According to the above process, changing the engine is necessary to dump ShaderCode; moreover, the disjointed workflow means that after dumping ShaderCode, you must invoke the zstd program to train the dictionary; finally, you still need to cook the project again to compress the Shader using the trained dictionary. Additionally, turning off Shader compression will cause DDC Cache Miss with LZ4 compression, leading to significant time overhead from repeated cooking, which is unacceptable in projects with a large number of Shaders.

Based on this pain point, I researched and implemented an efficient dictionary training method that requires no changes to the engine, allowing for rapid training of the dictionary and compression based on that dictionary. It can directly train the dictionary from ushaderbytecode and generate ushaderbytecode using ZSTD + dictionary compression, greatly improving processing efficiency. This is a completely Plugin-Only implementation, with almost zero integration cost, and will later be released as an extension module of HotPatcher.

Pak is a part of UFS in UE (Unreal File System), constructed as a virtual file system at the application layer. It is used to package game-related resources and files into a Pak file, avoiding the creation of a large number of file handles when accessing game resources at runtime, and allowing for read caching (PakCache) to enhance loading efficiency.

Moreover, within UFS, the priority of each Pak can be controlled to manage the priority of files in the file system. When loading files through UFS, files in Paks with higher priority will be accessed first, allowing them to replace files with lower priority. This is also key to UE’s implementation of hot updates, as detailed in the previous hot update series article .

However, by default, the packaging of Pak is done on the UE side, with PakUtilities and UnrealPak being developer-side functionalities that do not exist at runtime, meaning Pak files cannot be created during runtime. Nevertheless, Pak itself is in the form of an Archive, which theoretically can be reorganized at runtime.

This article will explore the implementation details and application directions of creating Pak files at runtime by discussing the aspects of Pak creation, file format, UFS analysis, and runtime reorganization feasibility.

As the scale of projects increases, the number of Shader variants in UE gradually accumulates, often reaching millions of Shader variants. Although UE provides the Share Material Shader Code feature to avoid duplicate storage of Shaders by serializing them into a standalone ushaderbytecode file, it can still occupy hundreds of megabytes of package size. By default, UE places these NotUAsset files in Chunk0, which must go into the base package, significantly affecting the base package for mobile platforms where hundreds of megabytes could hold many resources.

To solve this issue, we can only tackle it from three aspects:

1. Reduce the number of variants in the project, dump Shader information from the project, and analyze which ones are unnecessary;
2. Split shaderbytecode so that the base package only contains essential Shaders, while others are downloaded as needed. However, UE does not provide such a mechanism by default, but you can use my developed HotPatcher to split the base package, generate multiple shader libs, and dynamically download resources and required shaderbytecode using the hot update process.
3. Use a compression algorithm with a higher compression ratio to compress ShaderCode; the engine by default uses LZ4.

The first approach requires collaboration between TA and artists, making significant improvements relatively difficult. The second solution is detailed in previous articles about hot updates (Unreal Engine#热更新). This article starts from the third approach, implementing a special compression method for Shaders that effectively reduces the size of shaderbytecode, significantly improving the compression ratio of Shaders and can be combined with solution 2 to enhance compression while splitting shaderbytecode.

In the process of using UE development, specifying command line parameters is a frequently used feature that facilitates control over certain processes and toggles. UE also extensively uses Commandline to control the behavior of the engine. The official documentation: Command-Line Arguments, allows different modules to read or check parameters from the Commandline, implementing custom command line parameter functionalities.

However, for the mobile Android platform, it is not very convenient to specify startup parameters; editing the ue4commandline.txt file is quite cumbersome.

Based on this pain point, I developed a UE Android plugin during the New Year holiday that allows Android applications to start and specify command line parameters as conveniently as on PC, without needing to modify the engine. Once the plugin is activated, you can package the APK for use. The project is open-sourced on GitHub: hxhb/AppCmderUE.

This article analyzes how UE reads Commandline on the Android side and introduces the implementation principles and usage methods of the AppCmderUE plugin.

By default, when packaging a project in UE, BuildCookRun is invoked to execute a series of processes such as Compile/Cook/Pak/Stage. In UE, only the assets involved in Cook will be packaged; however, it often includes many unexpected assets, which can be perplexing. What resources does the engine depend on? And how should we manage the resources involved in packaging in UE?

This article starts with analyzing the rules for resource Cook during UE packaging, studying which resources will actually be Cooked, which is beneficial for resource management. Based on this understanding, a custom Cook process can be implemented, distributing Cook tasks to different processes or even machines to achieve parallelization and accelerate the UE build process.

In addition to resources like uasset, there are many Non-Asset files during packaging, such as ini, Shader Library, AssetRegistry, or script files added to the project, etc. These have been introduced in a previous article UE Hot Update: Demand Analysis and Solution Design. UE’s collection of these resources does not occur during the Cook phase (Shader Library and AssetRegistry are generated during the Cook phase), which will not be discussed further in this article, but a dedicated article will be written later on.

I have always hoped that everything I want to do can be quantified and recorded. Otherwise, as the years go by, I won’t remember what I have done. So I will make an annual summary.

As 2021 is coming to an end, reflecting on this year, many things happened, and I had many new ideas. It was also the year that had the greatest impact on me.

Recently, I checked the access data of the blog Google Analysic, and through traffic analysis, I found some very interesting points that can reflect, to some extent, the user characteristics in game development using Unreal Engine. The data sample in this article is the access traffic from our site for the week of 2021.11.14-2021.11.20. The overall data amount is limited and may not truly reflect real trends, only for reference.

The preview video for UE5 introduces a modular development mechanism for GamePlay, similar to mod-style development and management of game features and resources, referred to as “Game Feature”. This has already been enabled in UE4.27 and UE5. I think this new modular gameplay format is great, so I’ve conducted a technical research.

This article introduces the activation process and operational mechanism of Game Features, and at the end shares a demo based on Game Features. Additionally, I implemented a mechanism in HotPatcher that allows features to be packaged independently; Game Features do not need to be pre-packaged into the base package and can be downloaded and loaded on demand during runtime, achieving true modular loading.

Currently, the UE5 EA version still has some imperfections, and related content will be supplemented as the engine updates.

Usually, when developing projects using UE, there are multiple engine versions locally, and the Epic Game Launcher only supports launching installed engine versions and does not support source-compiled engines. When there are multiple engine versions locally, switching becomes inconvenient, and there is no streamlined tool to launch the engine, projects, and add launch parameters easily. When executing Commandlet, it requires creating many scripts, making management very cumbersome. Based on these pain points, I developed a UE launcher: UELauncher, to solve these issues, supporting both UE4 and UE5.

For the resource inspection requirements in the project, it needs to be simple and convenient to configure, automate execution, and be able to promptly locate related personnel. Based on this requirement, I have open-sourced a resource compliance scanning tool ResScannerUE, with configuration documentation: UE Resource Compliance Inspection Tool ResScannerUE.
This article will introduce how to achieve automated resource scanning through this tool, provide support for incremental detection combined with Git, use Commandlet to automatically trigger and execute detection rules for Content content changes on CI platforms, and be able to locate the most recent committers of problematic resources, achieving precise positioning and real-time scanning report sending to corporate WeChat, reminding relevant personnel to take action.
Additionally, I also provide a Git-based Pre-Commit Hook implementation, allowing detection of non-compliant resources before submitting and prohibiting the submission, thus avoiding the contamination of remote repositories with problematic resources. The overall solution has been meticulously designed and extensively optimized for experience and enhanced automation support, making it very convenient to configure and integrate, capable of meeting various resource scanning needs.