I previously wrote an article introducing UE’s default unpacking method (UE Hot Update: Splitting the Base Package), but the default unpacking method in UE is not flexible enough for large-scale resource projects and cannot meet the refined management needs of super-large-scale resource projects.
To address the issues with UE’s default packaging and resource splitting pain points, I developed an extension based on HotPatcherCore that can solve the drawbacks of the default unpacking process, making it sufficiently flexible and powerful. There’s an article on the blog that introduces its overall implementation mechanism: A Flexible and Non-Intrusive Base Package Splitting Solution.
HotChunker can be integrated into UE’s default packaging process non-intrusively, requiring no manual handling. As long as the default UE packaging is done, it will automatically trigger HotChunker’s unpacking process, achieving resource cooking, packaging, and automatic copying to the StagedBuilds directory.
This article will focus on how to configure resource splitting using HotChunker, automate builds, minimize resource redundancy, and implement parallel packaging.
Issues with UE’s Default Packaging
The CookerOnTheFly - UnrealPak mode used by UE has some inherent issues:
- Cannot precisely package and flexibly control custom unpacking behaviors
- No cross-platform universal pak entry filtering mechanism (only Android’s
ObbFilter) - Strong dependency on the Cook process, unable to package a single Chunk independently
- Difficult to perform programmatic unpacking conveniently
- Cannot configure Chunks to include Not-uasset files
- All Not-uasset files are in Chunk 0
- Unable to split Metadatas based on Chunks
The implementation of HotChunker can effectively address the above issues, making each Chunk self-contained:
The final effect for UE’s packaged resource configuration allows only the most basic resources to be managed and packaged by UE, while the rest can be split through HotChunker’s configuration.
The following part of the article will discuss how to configure this.
Chunk Configuration
The configuration of Chunks is the smallest unit for package splitting. In UE, each Chunk will be packaged into a separate pak file.
For HotChunker implementations, the configuration of a single Chunk can include the following attributes:
The resource configuration of a Chunk provides two modes: Rule/Getter, used to specify the resource path for packaging or programmatically obtain it from code.
Each Chunk can add external files as well as control the serialization of shaders, asset registry serialization, etc. Moreover, the Chunk’s metadata is self-contained, most notably reducing the size of the Shader Library within the installation package.
By default, the shaders of a UE packaged project are serialized into the same file, which is included in pakchunk0, meaning that the complete Shader of the entire project is within the installation package. As the resource scale increases, the size of the Shader Library can become extremely large, potentially reaching hundreds of MB. Making each Chunk’s Shader self-contained can effectively mitigate this issue. Coupled with my previous Efficient ZSTD Shader Dictionary Training Solution, it maximizes the reduction of Shader Library size and enables on-demand downloading of shaders.
Rule
The Rule mode configuration is similar to that of HotPatcher, allowing specification of included paths, resources, and ignored resources.
This mode is suitable for configuring relatively fixed resources, such as packaging a specific directory or a particular map.
Getter
In addition to directly specifying resources, I have also provided a Getter configuration mode that allows you to specify a class inheriting from UHPLGetter to dynamically obtain resources managed by the Chunk from code.
This mode was designed to establish a direct correspondence with the configurations by planners. It allows planning-based configurations to control packaging.
For example, the current version includes which maps and character classes, etc. These configurations will change continuously with the development process, and the data source is with the planners. In UE’s default method, modifications to packaging configurations still need to be made within the project.
However, HotChunker establishes a correspondence between packaging and planning configurations through this Getter configuration method, enabling planners to directly control which resources are packaged.
Chunk Management
HPL Resources
Similar to PrimaryLabel, I added an equivalent configuration for Chunk which I call HPL (HotPatcherPrimaryLabel).
You can create an HPL resource to edit the resources to be packaged; each HPL is an independent Chunk.
During packaging, you need to add the scan paths in Project Settings - Plugins - HotChunker:
In HotChunker packaging, it will automatically scan all configured directories for HPL resources for packaging.
DataTable
If you want to create multiple Chunks, you can create multiple HPL resources, but for easier management, I also support DataTable configuration.
Select to create a DataTable, and choose HPLChunkSetting for RoleStructure:
Open the editor, each element is an independent HPL Chunk configuration:
Additionally, I have added a dedicated packaging button for the HPL DataTable, allowing you to directly select which Chunk to package from the table interface:
It will trigger a separate process to execute without blocking the current editor.
Similarly, when a HPL DataTable is created and needs to be included in packaging, you need to add the ImportRuleTables configuration in Project Settings - Plugins - HotChunker:
Dynamic Registration
To achieve programmatic automated sub-packaging, I have added a dynamic registration mechanism for the splitting configuration. Simply bind the Delegate:
1 | FHotChunkerDelegates::OnNotyfyChunkRegister.AddStatic(&FGameEditorModule::OnNotyfyChunkRegister); |
In this function, register the Chunk:
Through dynamic registration or the previously mentioned Getter mechanism, the planning configuration can be correlated with the packaging, allowing you to automatically control which resources to package according to the planning configuration, liberating programmers from configuration tasks.
Editor Support
I provide editor options for Chunk’s DT configuration, conveniently allowing the selection of a specific Chunk for packaging. During development, artists can easily package the resources they need, avoiding wait times for packaging.
- Package a single Chunk
- Package all Chunks in the DT
- Import/Export Chunk configurations
- Non-blocking editor
Global Configuration and Process Extension
In Project Settings - Plugins - HotChunker, global configurations and logical replacements (ChunkerManager) can be made.
- HPL scan path
- Chunk Rule DT configuration
- Pak Setting
- Feature switches
- Replace ChunkerManager
- Global Overriding
Intervening at Various Stages of Chunker
To achieve sufficient flexibility, during implementation, I left many points open for intervention at various stages of Chunk packaging.
- Resource analysis and processing on demand
- Retrieve packaging status and results
- Get complete Manifest information
- Perform package audits
Within the framework, various analysis requirements of the project can be implemented non-intrusively.
Redundancy Control
I introduced the situation where priority does not take effect when packaging through UE’s default unpacking form in UE Hot Update: Splitting the Base Package#Priority Packaging Failure and provided a solution.
HotChunker has avoided these issues during its implementation, and Chunk configurations also support Priority.
If the priority of the Chunks is at the same level, then there will be no dependency between them, and some resources may become redundant.
Chunks included in the installation package do not have resource redundancy, while Chunks outside the installation package can control redundancy based on priority.
Commandlet Support
I have added various Commandlet supports to HotChunker, allowing for easy command line packaging.
- Package all configured Chunks in the project
- Package a group of Chunks
- Package a single Chunk
- Package a named Chunk
1 | -run=HotChunker -TargetPlatform=IOS |
Packaging Control and Dynamic Switches
Chunk packaging control is divided into three types:
- bUseHPL: Whether to enable HotChunker’s configuration for packaging Chunks
- ALL: All Chunks
- BASEBUILDS: Chunks within the installation package
- EXTERNAL: Chunks outside the installation package
In project settings, you can control the default packaging type:
Apart from fixed configurations, you can also pass commands in the commandlet to override configurations in the engine, which can facilitate controlling CI/CD.
Currently, there are two dynamic switches supported:
1 | # true/false |
When passing these during the launching of UE’s CookCommandlet, it will automatically control the packaging switches of the Chunker and the types of Chunks generated.
If you call HotChunkerCommandlet in your CI/CD, you can also pass these two dynamic switches.
For example, to package only non-installation package Chunks:
1 | -run=HotChunker -TargetPlatform=IOS -GenerateType=EXTERNAL |
This allows for easy integration into CI/CD systems:
Through this mechanism, precise control during packaging can be achieved. In certain testing scenarios, it enables rapid packaging, avoiding the need to process complete resources each time.
Chunk Encryption
Chunks managed and packaged using HotChunker default to reusing the DefaultCrypto.ini configuration in the project settings.
Configuration can be found in Project Settings - Plugins - HotChunker - HPLPakSettings - EncryptSettings:
bUseDefaultCryptoIni: Uses the configuration in project settings. If not reused, the CryptoKeys can specify a crypto.json for encryption.
Universal Package Filtering Solution
In the article UE Hot Update: Splitting the Base Package, I introduced methods for package filtering for Android and IOS.
The engine’s default only provides ObbFilter for Android without an equivalent for IOS, which needs to be implemented independently.
In the implementation of HotChunker, I supplemented the package filtering mechanism, allowing each Chunk to specify whether it should enter the installation package directly.
Moreover, special handling can be done for specific platforms: for instance, it can enter the base package by default, but not on the WindowsNoEditor platform.
If you prefer not to configure each Chunk individually, you can set a global override in project settings:
This configuration means that when packaging for the WindowsNoEditor platform, all managed chunks by HotChunker will be archived into the installation package.
Process Optimization
By default, UE uses the CookerOnTheFly process, which requires all resources involved in the packaging from the current project to be cooked first before it enters the Pak packaging process.
Essentially, it is a serial process:
However, when the scale of resources becomes large, this process can take a long time.
In reality, we do not necessarily need to complete the processing of all resources to create the installation package. Because, in mobile games, the resources in the installation package are only a small portion; the majority are dynamically downloaded during hot updates.
In HotChunker’s implementation, the packaging of installation package Chunks and the dynamic downloading of Chunks can be executed in parallel:
This can save considerable time, allowing the fastest possible speed to create an installation package and performing another separate packaging task in parallel for dynamic downloaded Chunks.
Conclusion
The packaging management solution based on HotChunker is a powerful and precise package management mechanism. It enables precise resource splitting while ensuring configuration flexibility and supports powerful Commandlet capabilities. It allows for easy non-intrusive process extension and resource auditing.
Moreover, this solution integrates non-intrusively, requiring no modifications to the default packaging process or the engine. The entire implementation is plugin-based, automatically executing when the plugin is enabled and the default UE packaging is triggered.
