Z's Blog

During the process of project development using Unreal Engine, various types of plugins are often developed and integrated to extend the engine, thereby fulfilling different requirements. For developers, understanding the operational mechanism is more important than using the tools. Thus, it is necessary to understand the principles behind the integration and development of plugins.

Previously, I also developed some tools and plugins for UE, and I hope to write a series of related articles summarizing the common technical content related to UE plugin and tool development and sharing some of my thoughts and functional scaffolding during plugin development. The goal is to achieve the desired functionality with minimal code, minimal invasiveness, and the best implementation strategies.

With the development of HotPatcher and my research on resource management technology, more and more features are being implemented around HotPatcher, and the implementation is becoming increasingly complex. Coupled with continuous updates of the engine, maintaining compatibility for a large and comprehensive plugin has become more difficult.

With the arrival of the new generation engine UE5, HotPatcher will also follow up with support for UE5 and develop new features based on UE5’s characteristics. I hope it will not only be a resource packaging tool but also include a comprehensive resource management solution encompassing resource management, auditing, packaging, hot updates, package optimization, and build enhancements, and serve as an open resource processing framework.

This article will introduce the modular progress of HotPatcher, support for existing features, and how to use the powerful packaging capability of HotPatcher to customize modular extensions based on project needs. Finally, I will discuss the future development plans for the HotPatcher project.

The Unreal Open Day 2022 was hosted by Epic in the form of an offline event combined with online live streaming. I am very happy to have attended the offline Unreal Open Day event this year. I recorded an online technical presentation and also participated in the live event in Shanghai. I am honored to have once again received the Outstanding Community Contributor award granted by Epic, which is both recognition and motivation for me. I look forward to the UE tech community continuing to grow and becoming the most vibrant game developer community. This article serves as a simple record, organizing the materials from UOD 2022, summarizing my participation in UOD, my presentation PPT, and related materials, as well as some photos taken at the UOD live event.

UE’s default resource management is relatively complex. By default, the resource packaging process is executed based on the configurations of maps, directories, and PrimaryAsset specified in ProjectSetting, as well as a combination of condition checks. Furthermore, UE’s Cook dynamically adds resources to the package based on runtime loading, making the resource packaging process almost a black box.

The blog introduces articles on default resource packaging rules and basic package splitting:

• UE Resource Management: Engine Packaging Resource Analysis
• UE Hot Update: Splitting Basic Packages

This article presents a new idea, utilizing HotPatcher‘s precise Cook and packaging mechanism to implement a HotChunker Mod that can perform non-intrusive, direct reuse of UE’s default packaging process and clearly split the basic package. This article will specifically introduce the usage and implementation principles.

Technical articles are an effective way to enhance expression skills and carry out technical accumulation. Unlike simple technical notes, I believe a technical article should be a solution to real problems or new technical ideas, rather than a mere assemblage of technical points.

Regarding how to write a qualified technical article, I have been contemplating this recently. This article will explore some of my thoughts and steps when writing articles, as well as tool recommendations and the implementation of CI/CD automation publishing.

ASTC stands for Adaptive Scalable Texture Compression, which is a popular texture compression scheme on mobile platforms. When the platform uses ASTC packaging, UE defaults to using the Intel ISPC Texture Compressor for texture compression. However, it has some limitations, only supporting compression specifications of 8x8 and above, while 10x10 and 12x12 are not supported. If specified in the project, it will still use the 8x8 specification. In addition to ISPC, the engine also provides ARM’s astc-encoder compression method, which supports specifications below 8x8, but is not enabled by default. Moreover, the integrated compression efficiency in the engine is very low, and using astc-encoder to compress textures can pose a significant challenge to Cook time in large-scale resources.

This article analyzes the settings for using ASTC compression for textures in UE, as well as the compression efficiency and optimization ideas for the astc-encoder in the engine.

In large projects, the scale of resources is immense, and the production teams involved are very diverse, including scenes, characters, UI, animations, effects, blueprints, data tables, and so on. Consequently, managing the volume and specification of resources becomes difficult to control.

For the established resource specifications, art production personnel may struggle to cover 100% of the scenarios, potentially overlooking details unintentionally. In most cases, issues are discovered after the packages are created, and for existing resources, a significant amount of manpower is required for handling them, making review and repair challenging.

Based on this pain point, I previously developed a resource scanning tool that allows convenient editing of rules to scan resources within the project.

• Creating Attribute Caches for Assets in UE using Reflection
• Design and Implementation Plan for Resource Self-Correction in UE
• Multi-Stage Automated Resource Check Plan in UE
• Automated Resource Checking Practice Based on ResScannerUE
• Resource Compliance Checking Tool ResScannerUE in UE

Recently, I have conducted a comprehensive upgrade of the plugin, enhancing its capabilities during editing and automated checks. This article will introduce how to utilize ResScannerUE to perform resource scans during editing, submitting, CI timed or Hook tasks, and Cooking phases, aiming to expose and prompt solutions to problematic resources as early as possible in production, thus avoiding anomalies in package resources.

In terms of specific implementation, many optimizations have been made for scanning speed, transforming the checking process into a nearly unnoticed action, which will be elaborated upon in the article.

In the article ZSTD Dictionary-based Shader Compression Scheme, I introduced a method for compressing UE’s ShaderCode using ZSTD dictionaries, which can significantly improve the compression ratio of ShaderLibrary. However, the process of training the dictionary and using it for compression remains complex and is not an efficient engineering implementation:

1. The engine’s default Shader compression must be turned off.
2. You need to cook the project once to dump the ShaderCode of each unique Shader.
3. Based on the dumped ShaderCode files, use the ZSTD program to train the dictionary.
4. Execute a full cook process again to compress using the dictionary and generate the final shaderbytecode.

According to the above process, changing the engine is necessary to dump ShaderCode; moreover, the disjointed workflow means that after dumping ShaderCode, you must invoke the zstd program to train the dictionary; finally, you still need to cook the project again to compress the Shader using the trained dictionary. Additionally, turning off Shader compression will cause DDC Cache Miss with LZ4 compression, leading to significant time overhead from repeated cooking, which is unacceptable in projects with a large number of Shaders.

Based on this pain point, I researched and implemented an efficient dictionary training method that requires no changes to the engine, allowing for rapid training of the dictionary and compression based on that dictionary. It can directly train the dictionary from ushaderbytecode and generate ushaderbytecode using ZSTD + dictionary compression, greatly improving processing efficiency. This is a completely Plugin-Only implementation, with almost zero integration cost, and will later be released as an extension module of HotPatcher.

Pak is a part of UFS in UE (Unreal File System), constructed as a virtual file system at the application layer. It is used to package game-related resources and files into a Pak file, avoiding the creation of a large number of file handles when accessing game resources at runtime, and allowing for read caching (PakCache) to enhance loading efficiency.

Moreover, within UFS, the priority of each Pak can be controlled to manage the priority of files in the file system. When loading files through UFS, files in Paks with higher priority will be accessed first, allowing them to replace files with lower priority. This is also key to UE’s implementation of hot updates, as detailed in the previous hot update series article .

However, by default, the packaging of Pak is done on the UE side, with PakUtilities and UnrealPak being developer-side functionalities that do not exist at runtime, meaning Pak files cannot be created during runtime. Nevertheless, Pak itself is in the form of an Archive, which theoretically can be reorganized at runtime.

This article will explore the implementation details and application directions of creating Pak files at runtime by discussing the aspects of Pak creation, file format, UFS analysis, and runtime reorganization feasibility.

As the scale of projects increases, the number of Shader variants in UE gradually accumulates, often reaching millions of Shader variants. Although UE provides the Share Material Shader Code feature to avoid duplicate storage of Shaders by serializing them into a standalone ushaderbytecode file, it can still occupy hundreds of megabytes of package size. By default, UE places these NotUAsset files in Chunk0, which must go into the base package, significantly affecting the base package for mobile platforms where hundreds of megabytes could hold many resources.

To solve this issue, we can only tackle it from three aspects:

1. Reduce the number of variants in the project, dump Shader information from the project, and analyze which ones are unnecessary;
2. Split shaderbytecode so that the base package only contains essential Shaders, while others are downloaded as needed. However, UE does not provide such a mechanism by default, but you can use my developed HotPatcher to split the base package, generate multiple shader libs, and dynamically download resources and required shaderbytecode using the hot update process.
3. Use a compression algorithm with a higher compression ratio to compress ShaderCode; the engine by default uses LZ4.

The first approach requires collaboration between TA and artists, making significant improvements relatively difficult. The second solution is detailed in previous articles about hot updates (Unreal Engine#热更新). This article starts from the third approach, implementing a special compression method for Shaders that effectively reduces the size of shaderbytecode, significantly improving the compression ratio of Shaders and can be combined with solution 2 to enhance compression while splitting shaderbytecode.