Z's Blog

When using the Unreal Engine development tool, a significant number of tasks involve resource processing and data export needs. These tasks need to be performed frequently and automatically, often integrated into a CI/CD system.

In the specific implementation, the Commandlet mechanism of UE is utilized to drive the engine in a command-line manner, performing custom behaviors.

Taking the Commandlet features supported in the plugin I developed as an example:

1. HotPatcher: Export basic package information, pack patches
2. ResScannerUE: Incremental scanning of changed resources
3. HotChunker: Standalone packaging of Chunk
4. libZSTD: Training Shader dictionaries
5. ExportNavMesh: Export NavMesh data

Commandlet allows for easier integration into CI/CD to achieve automation.

In this article, I will primarily introduce the Commandlet mechanism of UE, analyze its implementation principles, and provide some development tips and insights from my development process.

Additionally, this is the second article in my UE Plugin and Tool Development series, which will continue to be updated, so stay tuned.

I previously wrote an article introducing UE’s default unpacking method (UE Hot Update: Splitting the Base Package), but the default unpacking method in UE is not flexible enough for large-scale resource projects and cannot meet the refined management needs of super-large-scale resource projects.

To address the issues with UE’s default packaging and resource splitting pain points, I developed an extension based on HotPatcherCore that can solve the drawbacks of the default unpacking process, making it sufficiently flexible and powerful. There’s an article on the blog that introduces its overall implementation mechanism: A Flexible and Non-Intrusive Base Package Splitting Solution.

HotChunker can be integrated into UE’s default packaging process non-intrusively, requiring no manual handling. As long as the default UE packaging is done, it will automatically trigger HotChunker’s unpacking process, achieving resource cooking, packaging, and automatic copying to the StagedBuilds directory.

This article will focus on how to configure resource splitting using HotChunker, automate builds, minimize resource redundancy, and implement parallel packaging.

Memory optimization is a frequent focus in game development. To avoid excessive memory allocation that triggers an OOM (Out of Memory) to be forcibly terminated by the system, common optimization methods typically start from the usage perspective, enhancing memory utilization efficiency by trimming unnecessary features and controlling resource loading.

However, there is another approach to increase the threshold for triggering OOM, allowing the system to permit our app to allocate more memory. In the new iOS versions, Apple has introduced new memory features for apps, allowing them to extend addressable space and increase allocatable memory.

In this article, I will explore how to leverage these features in UE to enhance the total allocatable memory for games on the iOS platform.

During the process of project development using Unreal Engine, various types of plugins are often developed and integrated to extend the engine, thereby fulfilling different requirements. For developers, understanding the operational mechanism is more important than using the tools. Thus, it is necessary to understand the principles behind the integration and development of plugins.

Previously, I also developed some tools and plugins for UE, and I hope to write a series of related articles summarizing the common technical content related to UE plugin and tool development and sharing some of my thoughts and functional scaffolding during plugin development. The goal is to achieve the desired functionality with minimal code, minimal invasiveness, and the best implementation strategies.

With the development of HotPatcher and my research on resource management technology, more and more features are being implemented around HotPatcher, and the implementation is becoming increasingly complex. Coupled with continuous updates of the engine, maintaining compatibility for a large and comprehensive plugin has become more difficult.

With the arrival of the new generation engine UE5, HotPatcher will also follow up with support for UE5 and develop new features based on UE5’s characteristics. I hope it will not only be a resource packaging tool but also include a comprehensive resource management solution encompassing resource management, auditing, packaging, hot updates, package optimization, and build enhancements, and serve as an open resource processing framework.

This article will introduce the modular progress of HotPatcher, support for existing features, and how to use the powerful packaging capability of HotPatcher to customize modular extensions based on project needs. Finally, I will discuss the future development plans for the HotPatcher project.

The Unreal Open Day 2022 was hosted by Epic in the form of an offline event combined with online live streaming. I am very happy to have attended the offline Unreal Open Day event this year. I recorded an online technical presentation and also participated in the live event in Shanghai. I am honored to have once again received the Outstanding Community Contributor award granted by Epic, which is both recognition and motivation for me. I look forward to the UE tech community continuing to grow and becoming the most vibrant game developer community. This article serves as a simple record, organizing the materials from UOD 2022, summarizing my participation in UOD, my presentation PPT, and related materials, as well as some photos taken at the UOD live event.

UE’s default resource management is relatively complex. By default, the resource packaging process is executed based on the configurations of maps, directories, and PrimaryAsset specified in ProjectSetting, as well as a combination of condition checks. Furthermore, UE’s Cook dynamically adds resources to the package based on runtime loading, making the resource packaging process almost a black box.

The blog introduces articles on default resource packaging rules and basic package splitting:

• UE Resource Management: Engine Packaging Resource Analysis
• UE Hot Update: Splitting Basic Packages

This article presents a new idea, utilizing HotPatcher‘s precise Cook and packaging mechanism to implement a HotChunker Mod that can perform non-intrusive, direct reuse of UE’s default packaging process and clearly split the basic package. This article will specifically introduce the usage and implementation principles.

Technical articles are an effective way to enhance expression skills and carry out technical accumulation. Unlike simple technical notes, I believe a technical article should be a solution to real problems or new technical ideas, rather than a mere assemblage of technical points.

Regarding how to write a qualified technical article, I have been contemplating this recently. This article will explore some of my thoughts and steps when writing articles, as well as tool recommendations and the implementation of CI/CD automation publishing.

ASTC stands for Adaptive Scalable Texture Compression, which is a popular texture compression scheme on mobile platforms. When the platform uses ASTC packaging, UE defaults to using the Intel ISPC Texture Compressor for texture compression. However, it has some limitations, only supporting compression specifications of 8x8 and above, while 10x10 and 12x12 are not supported. If specified in the project, it will still use the 8x8 specification. In addition to ISPC, the engine also provides ARM’s astc-encoder compression method, which supports specifications below 8x8, but is not enabled by default. Moreover, the integrated compression efficiency in the engine is very low, and using astc-encoder to compress textures can pose a significant challenge to Cook time in large-scale resources.

This article analyzes the settings for using ASTC compression for textures in UE, as well as the compression efficiency and optimization ideas for the astc-encoder in the engine.

In large projects, the scale of resources is immense, and the production teams involved are very diverse, including scenes, characters, UI, animations, effects, blueprints, data tables, and so on. Consequently, managing the volume and specification of resources becomes difficult to control.

For the established resource specifications, art production personnel may struggle to cover 100% of the scenarios, potentially overlooking details unintentionally. In most cases, issues are discovered after the packages are created, and for existing resources, a significant amount of manpower is required for handling them, making review and repair challenging.

Based on this pain point, I previously developed a resource scanning tool that allows convenient editing of rules to scan resources within the project.

• Creating Attribute Caches for Assets in UE using Reflection
• Design and Implementation Plan for Resource Self-Correction in UE
• Multi-Stage Automated Resource Check Plan in UE
• Automated Resource Checking Practice Based on ResScannerUE
• Resource Compliance Checking Tool ResScannerUE in UE

Recently, I have conducted a comprehensive upgrade of the plugin, enhancing its capabilities during editing and automated checks. This article will introduce how to utilize ResScannerUE to perform resource scans during editing, submitting, CI timed or Hook tasks, and Cooking phases, aiming to expose and prompt solutions to problematic resources as early as possible in production, thus avoiding anomalies in package resources.

In terms of specific implementation, many optimizations have been made for scanning speed, transforming the checking process into a nearly unnoticed action, which will be elaborated upon in the article.