UE热更新：资产管理与审计工具

In the previous article, the rules and implementation of basic package splitting were introduced. After the basic packaging rules are stabilized, the focus in daily development shifts to asset management, package resource auditing, and analyzing asset sizes and redundancy in the project.

This article introduces resource packaging configuration in UE, commonly used asset management methods, and asset auditing tools in engineering practice. It is also a supplement to the previous article UE4 Hot Update: Splitting Basic Packages in the series on resource management for hot updates.

Packaging Configuration

Resource packaging options in UE can be set in Project Settings - Packaging, where you can configure whether to generate Pak files and the rules for including resources in the package.

Regarding the specific content included in the UE4 basic package, I have introduced it in the Unreal Open Day hot update plan. If you’re interested, you can watch this video: Unreal Open Day 2020 Unreal Engine 4 Cross-Platform Hot Update Plan | Charlie Peng.

UsePakFile: Whether to use Pak files. If enabled, all resources used by the game will be packaged into a pak file. If disabled, the cooked uasset/config/shaderbytecode/assetregistry/uplugin/uproject files will be stored individually according to their path rules within the project.

Generate Chunks: Whether to generate chunks during packaging. If there are rules for chunk division in the project, checking this option will generate multiple pak files.
Generate No Chunks: Do not generate chunks.
Chunk Hard Reference Only: Only package resources that are hard referenced in the chunk; soft references will not be included.
PakFile Compression Formats: Specify the compression algorithm for Pak files. This parameter will be passed to UnrealPak. See: ModularFeature: Integrating ZSTD Compression Algorithm for UE4.
ShareMaterialShaderCode: Whether to share ShaderCode. It generates ushaderbytecode into the package, which can reduce the package size.

The following parameters specifically control the rules for packaging resources:

Directories listed in Directories to never cook will not be included in the package.

Additional Non-Asset Directories to Package is used to package some non-uasset files, such as Lua or other types of table files.

Chunk Division

UE has the functionality to divide chunks. Each chunk will generate a separate pak file during packaging, which can be configured by creating a PrimaryAssetLabel.

Documentation on chunk division in UE:

Creating a PrimaryAssetLabel resource:

Note: By default, engine resources and non-asset resources configured in project settings are forcibly packaged into Chunk 0. If engine resources are included, they will not be packaged into the current chunk (setting Priority has no effect). The PrimaryAssetLabel can only control the division of uasset assets.

The relevant code is in:

Runtime/Engine/Private/AssetManager.cpp
bool UAssetManager::GetPackageChunkIds(FName PackageName, const ITargetPlatform* TargetPlatform, const TArray<int32>& ExistingChunkList, TArray<int32>& OutChunkList, TArray<int32>* OutOverrideChunkList) const
{
    // Include preset chunks
    OutChunkList.Append(ExistingChunkList);
    if (OutOverrideChunkList)
    {
        OutOverrideChunkList->Append(ExistingChunkList);
    }

    if (PackageName.ToString().StartsWith(TEXT("/Engine/"), ESearchCase::CaseSensitive))
    {
        // Some engine content is only referenced by string, make sure it's all in chunk 0 to avoid issues
        OutChunkList.AddUnique(0);

        if (OutOverrideChunkList)
        {
            OutOverrideChunkList->AddUnique(0);
        }
    }

    // Add all chunk ids from the asset rules of managers. By default priority will not override other chunks
    TSet<FPrimaryAssetId> Managers;
    Managers.Reserve(128);

    GetPackageManagers(PackageName, true, Managers);
    return GetPrimaryAssetSetChunkIds(Managers, TargetPlatform, ExistingChunkList, OutChunkList);
}

By editing the parameters of the created PrimaryAssetLabel resource, we can control the resources managed by it in the chunk:

Priority: Specifies the priority of the current chunk. Since multiple chunks may contain the same resource, how to decide which chunk this resource should be packaged into? By specifying the priority, you can control asset redundancy. If priorities are the same, each chunk will contain one copy.
ChunkID: Specifies the ID value for the generated chunk. The naming rule for the generated pak file is: pakchunk0-WindowsNoEditor.pak, where the number after pakchunk is the ChunkID value.
ApplyRecursively: Perform recursive dependency analysis on the managed resources, but not include assets managed by higher-priority chunks.
CookRule: The Cook rules for the current chunk, which can control the resources managed by this PrimaryAssetLabel to not be packaged (NeverCook)
- UnKnown: If referenced, then cook.
- NeverCook: Never cook; these resources will not be packaged.
- DevelopmentCook: Cook when referenced in Development mode, but not in Production.
- DevelopmentAlwaysCook: Always cook in Development mode; do not cook in Production.
- AlwaysCook: Cook in any case.
Label Assets in My Directory: All resources under the directory (and subdirectories) where the current PrimaryAssetLabel is located are managed by it.
IsRuntimeLabel: Sets the PrimaryAssetLabel resource itself to be available at runtime and included in the package. This option does not affect the assets it tags.
Explicit Assets: Specifies the resource files managed by it, similar to IncludeSpecifyAsset in HotPatcher.
ExplicitBlueprints: Specifies the management of a certain type of blueprint asset.
AssetCollection: Specifies the collection name.

From the parameters mentioned above, we can achieve the following asset management functionalities:

Specify a directory.
Specify resources and their dependencies.
Control not packaging certain parameters (Never Cook).
Control asset redundancy through chunk priority.

In project asset management, these modes can be combined to meet project needs:

If there are many map copies, the copies and their resources can be managed as a single chunk. If some resources are redundantly referenced in other chunks, those resources can be managed as a separate chunk and set to a higher priority.

In the resource preview (Reference Viewer), you can see resources directly referenced by the chunk. You need to check Show Management References. The type referenced by the chunk is EAssetRegistryDependencyType::Manage, not a normal Hard/Soft resource reference relationship.

However, PrimaryAssetLabel also has a drawback; you cannot specify non-uasset files. If we need to update Lua or databases, tables, etc., it cannot be packaged and divided using the PrimaryAssetLabel mechanism. This requirement can be fulfilled through HotPatcher, which also implements chunk functionality similar to PrimaryAssetLabel for splitting patches. For details, see the previous article: UE4 Resource Hot Update Packaging Tool HotPatcher.

When we use PrimaryAssetLabel and enable Generate Chunks in Project Settings - Packaging, multiple pak files will be generated according to the configured rules during packaging:

D:\PCVersion\WindowsNoEditor>tree /a /f
|   ClientGame.exe
|   Manifest_DebugFiles_Win64.txt
|   Manifest_NonUFSFiles_Win64.txt
|   
+---Engine
\---ClientGame
    +---Binaries
    |   \---Win64
    |           ClientGame.exe
    |           ClientGame.pdb
    |           
    +---Content 
    |   \---Paks
    |           pakchunk0-WindowsNoEditor.pak
    |           pakchunk1-WindowsNoEditor.pak
    |           pakchunk2-WindowsNoEditor.pak
    |           
    \---Plugins

The multiple chunks’ pak files created here can be set in the previous basic package splitting rules to achieve the functionality of basic package splitting.

Asset Auditing

Asset Audit

The previous two sections introduced project configuration for packaging in UE and methods for dividing assets into chunks. This section will discuss methods and tools for conducting asset audits in daily development.

Through Chunk Division, we can mark some resources for packaging into pak files. However, we cannot intuitively know how many resources are managed by this chunk, or their sizes on each platform, as UE requires cooking during packaging, and the final size of each resource is platform-dependent. Therefore, a method is needed to visually see the resource division within the basic package’s chunk.

UE provides an asset auditing tool: Asset Audit, which can be opened in the Editor via Windows - Developer Tools - Asset Audit.

You can see a series of information such as resource paths, sizes, and which chunks they are located in, making it easy to check resource sizes and redundancy in chunks.

It has already been mentioned that the resource sizes seen in the Editor and the sizes that end up in the package are different. In the upper right corner of the Asset Audit window, the platforms that have been packaged (Saved/Cooked) are listed.

The Asset Audit needs to read the DevelopmentAssetRegistry.bin file to obtain resource information for a platform. After packaging a platform, it is stored in the following path:

1	Client\Saved\Cooked\WindowsNoEditor\Client\Metadata\DevelopmentAssetRegistry.bin

This file records the size and reference relationships of resources after cooking for a platform. Note that the resources like Texture2D that are set for compression after cooking will execute compression when packaged into pak. The listed sizes are the original sizes of cooked resources without pak compression.

You can automatically extract the Metadata directory under the Cooked directory during packaging. In the AssetAudit window’s upper right corner, select Custom and then choose the DevelopmentAssetRegistry.bin file.

UnrealPakViewer

UnrealPakViewer is a tool for viewing pak files. It allows you to conveniently view file information in the pak and decompress files from the pak.

The author also supports specifying the loading of AssetRegistry files because, by default, AssetRistry.bin is only included in Chunk 0. Opening a pak that does not include AssetRegistry.bin will not allow previewing of resource types. You can use UnrealPakViewer by specifying an external file’s AssetRegistry.bin, which can be combined with AssetAudit as a daily asset auditing tool for your project.