Z's Blog

UE Resource Management: Engine Packaging Resource Analysis

UE资源管理：引擎打包资源分析
Posted on In , Views:
Symbols count in article: 9.8k Reading time ≈ 24 mins.

By default, when packaging a project in UE, BuildCookRun is invoked to execute a series of processes such as Compile/Cook/Pak/Stage. In UE, only the assets involved in Cook will be packaged; however, it often includes many unexpected assets, which can be perplexing. What resources does the engine depend on? And how should we manage the resources involved in packaging in UE?

This article starts with analyzing the rules for resource Cook during UE packaging, studying which resources will actually be Cooked, which is beneficial for resource management. Based on this understanding, a custom Cook process can be implemented, distributing Cook tasks to different processes or even machines to achieve parallelization and accelerate the UE build process.

In addition to resources like uasset, there are many Non-Asset files during packaging, such as ini, Shader Library, AssetRegistry, or script files added to the project, etc. These have been introduced in a previous article UE Hot Update: Demand Analysis and Solution Design. UE’s collection of these resources does not occur during the Cook phase (Shader Library and AssetRegistry are generated during the Cook phase), which will not be discussed further in this article, but a dedicated article will be written later on.

The resources involved in Cook in UE can be logically divided into the following categories:

  1. Key resources configured in project settings, such as StartupMap, GameMode, GameInstance, DefaultTouchInterface, etc., which are essential resources.
  2. Several UI directories packaged by default.
  3. Resources loaded via code during engine startup.
  4. Cook resources configured in project settings (including resources marked with Directory to Always Cook, PrimaryAssetLabel, etc.).
  5. Resources passed to CookOnTheFlyServer via FGameDelegates::Get().GetCookModificationDelegate().
  6. If no specific resources are specified under certain conditions, resources in project and plugin directories are analyzed.
  7. Localized resources (UE supports different resources for different cultures, though it is not commonly used).

The resource analysis process in UE is very complex and scattered across various places, and it contains various condition checks. Analyzing the project’s dependent resources accurately is quite inconvenient. This is why it is challenging to ascertain precisely what resources are packaged by UE.

Based on these pain points, I plan to implement a concise and standardized resource collection and packaging process based on HotPatcher. Essentially, the goal is to package the necessary resources that the engine and application need at runtime. This fundamental need leads to the complexity of resource configuration and detection in UE, but we can simplify it for unified analysis and management.

CookCommandlet

The engine provides UCookCommandlet to implement resource Cook, and during the packaging process, it is invoked by UAT. The default Cook command is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
D:/UnrealEngine/Engine/Engine/Binaries/Win64/UE4Editor-Cmd.exe
D:/UnrealProjects/Blank425/Blank425.uproject
-run=Cook
-TargetPlatform=WindowsNoEditor
-fileopenlog
-unversioned
-abslog=D:/UnrealEngine/Engine/Engine/Programs/AutomationTool/Saved/Cook-2021.12.07-15.47.10.txt
-stdout
-CrashForUAT
-unattended
-NoLogTimes
-UTF8Output

It executes to the main function of UCookCommandlet:

1
2
3
4
5
6
7
8
9
10
11
Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp
/* UCommandlet interface
 *****************************************************************************/

int32 UCookCommandlet::Main(const FString& CmdLineParams)
{
  COOK_STAT(double CookStartTime = FPlatformTime::Seconds());
  Params = CmdLineParams;
  ParseCommandLine(*Params, Tokens, Switches);
  // ...
}

After some parameter checks, it passes the execution flow to CookByTheBook, creates CookOnTheFlyServer, and calls StartCookByTheBook.

All resources during the engine’s packaging are Cooked in CookOnTheFlyServer, generating Shader and AssetRegistry. It can be said that CookOnTheFlyServer is the process in which UE packages assets in a common format in the editor, serializing them to platform format.

Resource Analysis

The thought process behind resource loading during UE packaging is as follows: first, find the local uasset files, convert the paths to PackageName, load them; while loading, dependent resources are also loaded and cooked together.

StartupPackages

In CookOnTheFlyServer.cpp, the resources already loaded into memory are added to CookByTheBookOptions->StartupPackages:

CookOnTheFlyServer will subsequently add them to the Cook list and handle redirectors.

AllMaps

If no maps are specified via the command line, AllMaps will be added to MapIniSections:


It is a section in DefaultEditor.ini:

1
2
3
4
5
DefaultEngine.ini
[AllMaps]
+Map=/Game/Maps/Login
+Map=/Game/Maps/LightSpeed
+Map=/Game/Maps/VFXTest

Before starting, a global compilation of GlobalShader occurs:

Resources are obtained via GRedirectCollector:

UI

By default, UE adds directories under ContentDirectories in BaseEditor.ini to the Cook list: Engine/Config/BaseEditor.ini#L271

The default configuration in the engine is as follows, and other directories can be added by modifying DefaultEditor.ini:

1
2
3
4
5
6
7
BaseEditor.ini
[UI]
; Directories specifying assets needed by Slate UI, assets in these directories are always cooked even if not referenced
+ContentDirectories=/Game/UI
+ContentDirectories=/Game/Widget
+ContentDirectories=/Game/Widgets
+ContentDirectories=/Engine/MobileResources

Resources in these directories will be packaged: CookOnTheFlyServer.cpp#L5519

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
//@todo SLATE: This is a hack to ensure all slate referenced assets get cooked.
// Slate needs to be refactored to properly identify required assets at cook time.
// Simply jamming everything in a given directory into the cook list is error-prone
// on many levels - assets not required getting cooked/shipped; assets not put under 
// the correct folder; etc.
if ( !(FilesToCookFlags & ECookByTheBookOptions::NoSlatePackages))
{
  TArray<FString> UIContentPaths;
  TSet <FName> ContentDirectoryAssets; 
  if (GConfig->GetArray(TEXT("UI"), TEXT("ContentDirectories"), UIContentPaths, GEditorIni) > 0)
  {
    for (int32 DirIdx = 0; DirIdx < UIContentPaths.Num(); DirIdx++)
    {
      FString ContentPath = FPackageName::LongPackageNameToFilename(UIContentPaths[DirIdx]);

      TArray<FString> Files;
      IFileManager::Get().FindFilesRecursive(Files, *ContentPath, *(FString(TEXT("*")) + FPackageName::GetAssetPackageExtension()), true, false);
      for (int32 Index = 0; Index < Files.Num(); Index++)
      {
        FString StdFile = Files[Index];
        FName PackageName = FName(*FPackageName::FilenameToLongPackageName(StdFile));
        ContentDirectoryAssets.Add(PackageName);
        FPaths::MakeStandardFilename(StdFile);
        AddFileToCook(FilesInPath, StdFile);
      }
    }
  }

  if (CookByTheBookOptions && CookByTheBookOptions->bGenerateDependenciesForMaps) 
  {
    for (auto& MapDependencyGraph : CookByTheBookOptions->MapDependencyGraphs)
    {
      MapDependencyGraph.Value.Add(FName(TEXT("ContentDirectoryAssets")), ContentDirectoryAssets);
    }
  }
}

If you do not want the resources under the /Game/UI directory to be included during default packaging, you can override this in the project configuration file DefaultEditor.ini:

1
2
3
[UI]
!ContentDirectories=ClearArray
+ContentDirectories=/Engine/MobileResources

Directory to Always Cook

Project Settings-Directory to Always Cook DirectoriesToAlwaysCook

Maps

1
2
3
DefaultGame.ini
[/Script/UnrealEd.ProjectPackagingSettings]
+MapsToCook=(FilePath="/Game/HandheldAR/Maps/HandheldARBlankMap")

Cultures

Here it does not refer to multiple languages but to different cultures, allowing for the use of different resources. Asset Localization

Code for obtaining culture resources: CookOnTheFlyServer.cpp#L6714

Resources are read from project settings:

It will traverse all RootPaths, such as /Engine, /Game, and root directories of plugins:

For example, resources under the following directory, and will recursively include subdirectories:

1
/Game/L10N/en/

DefaultTouchInterface

DefaultTouchInterface is a configurable virtual joystick class in the engine. It may not be depended on by other resources, but it still needs to be packaged; hence it’s individually retrieved during Cook:

1
2
3
4
5
6
7
8
9
10
FConfigFile InputIni;
FString InterfaceFile;
FConfigCacheIni::LoadLocalIniFile(InputIni, TEXT("Input"), true);
if (InputIni.GetString(TEXT("/Script/Engine.InputSettings"), TEXT("DefaultTouchInterface"), InterfaceFile))
{
  if (InterfaceFile != TEXT("None") && InterfaceFile != TEXT(""))
  {
    SoftObjectPaths.Emplace(InterfaceFile);
  }
}

GetCookModificationDelegate

Binding agents can be passed to CookCommandlet for the files to be Cooked:

1
2
3
// allow the game to fill out the asset registry, as well as get a list of objects to always cook
TArray<FString> FilesInPathStrings;
FGameDelegates::Get().GetCookModificationDelegate().ExecuteIfBound(FilesInPathStrings);

Note that the passed resources need to be the absolute paths of uasset files, not resource paths like /Game/xxx.

AssetManager

Accessing PrimaryAssetTypeInfo through UAssetManager::Get().ModifyCook (configured in project settings, PrimaryAssetLabelId, etc.).

The following two are the ones automatically added in new projects; ModifyCook scans all PrimaryAssetId resources and adds specified resources to PackageToCook. Note that only these resources are added.

If no resources are explicitly specified in the command line, and no resources are obtained from FGameDelegates::Get().GetCookModificationDelegate(), or UAssetManager::Get().Modify, all resources from plugins and the project will be added:

The execution criteria are:

  1. AlwaysCookMaps in DefaultEditor.ini is empty, AllMaps is empty.
  2. List of maps to include in a packaged build in project settings is empty.
  3. DirectoriesToAlwaysCook in project settings is empty.
  4. Resources obtained from FGameDelegates::Get().GetCookModificationDelegate() are empty.
  5. Resources obtained from UAssetManager::Get().ModifyCook are empty.

In this case, it will fetch /Engine, /Game, and all enabled plugin umap, uasset through NormalizePackageNames:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// If no packages were explicitly added by command line or game callback, add all maps
if (FilesInPath.Num() == InitialPackages.Num() || bCookAll)
{
  TArray<FString> Tokens;
  Tokens.Empty(2);
  Tokens.Add(FString("*") + FPackageName::GetAssetPackageExtension());
  Tokens.Add(FString("*") + FPackageName::GetMapPackageExtension());

  uint8 PackageFilter = NORMALIZE_DefaultFlags | NORMALIZE_ExcludeEnginePackages | NORMALIZE_ExcludeLocalizedPackages;
  if (bMapsOnly)
  {
    PackageFilter |= NORMALIZE_ExcludeContentPackages;
  }

  if (bNoDev)
  {
    PackageFilter |= NORMALIZE_ExcludeDeveloperPackages;
  }

  // assume the first token is the map wildcard/pathname
  TArray<FString> Unused;
  for (int32 TokenIndex = 0; TokenIndex < Tokens.Num(); TokenIndex++)
  {
    TArray<FString> TokenFiles;
    if (!NormalizePackageNames(Unused, TokenFiles, Tokens[TokenIndex], PackageFilter))
    {
      UE_LOG(LogCook, Display, TEXT("No packages found for parameter %i: '%s'"), TokenIndex, *Tokens[TokenIndex]);
      continue;
    }

    for (int32 TokenFileIndex = 0; TokenFileIndex < TokenFiles.Num(); ++TokenFileIndex)
    {
      AddFileToCook(FilesInPath, TokenFiles[TokenFileIndex]);
    }
  }
}

But with the default filters, it will exclude resources from the engine directory (/Engine) and from localization directories (/*/L10N/).

This essentially includes all uasset and umap from projects and plugins, excluding all resources under the L10N directory.

The core function for collecting packaged content is located in UCookOnTheFlyServer::CollectFilesToCook: CookOnTheFlyServer.cpp#L5200.

1
2
3
DefaultInput.ini
[/Script/Engine.InputSettings]
DefaultTouchInterface=/Engine/MobileResources/HUD/LeftVirtualJoystickOnly.LeftVirtualJoystickOnly

SkipEditorContent

You can configure this in project settings to ignore Editor-related resources during Cook:

In CookOnTheFlyServer, it will ignore resources from /Engine/Editor* and /Editor/VREditor*:

1
2
3
4
5
6
7
8
// don't save Editor resources from the Engine if the target doesn't have editoronly data
if (IsCookFlagSet(ECookInitializationFlags::SkipEditorContent) &&
	(PackagePathName.StartsWith(TEXT("/Engine/Editor")) || PackagePathName.StartsWith(TEXT("/Engine/VREditor"))) &&
	!Target->HasEditorOnlyData())
{
	Result = ESavePackageResult::ContainsEditorOnlyData;
	bCookPackage = false;
}

Dependency Loading During Cook

Although the above lists the resources that will be included during the engine’s packaging, they are not everything since they are still individual resources or directories and do not include the dependency relationships of the resources. Therefore, UE will also perform a substantial dependency analysis during its Cook process.

Consider the following two questions:

  1. If an Actor with a C++ implementation is placed in a map and its constructor code loads a certain resource, how will it be packaged?
1
2
3
4
5
6
7
AMyActor::AMyActor()
{
  // Set this actor to call Tick() every frame.  You can turn this off to improve performance if you don't need it.
  PrimaryActorTick.bCanEverTick = true;

  UTexture2D* Texture2D = LoadObject<UTexture2D>(nullptr, TEXT("/Game/TextureResources/T_ImportTexture.T_ImportTexture"));
}

Placing it in the scene will not generate any dependencies:

  1. How do we package resources that are not referenced in AssetRegistry dependency relationships? For example, AnimSequence configurations such as BoneCompressionSettings and CurveCompressionSettings do not appear in the dependency relationships and cannot be found when retrieving dependencies of the animation sequence from the AssetRegistry.

However, they are recorded in the uasset’s ImportTable:

From the Cooked uasset, it can also be seen:

Given this, why can’t we directly scan dependencies from the ImportTable of the assets?

This is because accessing the ImportTable requires actually loading the resource, which can be very resource-intensive and time-consuming with a large number of resources. Accessing dependency relationships from the AssetRegistry does not require loading resources into memory, making it much quicker.

So, how can we solve these issues? We need to start with the default implementation mechanism of UE. It’s essential to clarify two points:

  1. The engine creates CDO upon startup and executes the class constructor.
  2. UE’s resources will load their dependent resources when they are loaded.

Based on this thought, UE has implemented a solution during Cook to listen for all created UObjects and add them to the Cook list to ensure resources loaded in C++ constructors and dependent resources loaded through the ImportTable are also cooked.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
struct FPackageTracker : public FUObjectArray::FUObjectCreateListener, public FUObjectArray::FUObjectDeleteListener
{
  FPackageTracker()
  {
  	for (TObjectIterator<UPackage> It; It; ++It)
	{
		UPackage* Package = *It;
	
		if (Package->GetOuter() == nullptr)
		{
			LoadedPackages.Add(Package);
		}
	}
    GUObjectArray.AddUObjectDeleteListener(this);
    GUObjectArray.AddUObjectCreateListener(this);
  }
  ~FPackageTracker()
  {
    GUObjectArray.RemoveUObjectDeleteListener(this);
    GUObjectArray.RemoveUObjectCreateListener(this);
  }

  virtual void NotifyUObjectCreated(const class UObjectBase *Object, int32 Index) override
  {
    // ...
  }

  virtual void NotifyUObjectDeleted(const class UObjectBase *Object, int32 Index) override
  {
    // ...
  }
};

Thus, following the same thought process, we only need to use the AssetRegistry to get the dependencies of the resources, store a rough list of resources, and then during Cook, listen to the creation of UObjects. If they are not in the scanned resource list, we will add them to the Cook queue, thereby achieving a complete resource packaging process.

The comparison between the Cook results without importing the ImportTable (left being the default UE Cook results, right being the custom Cook results):
未追踪ImportTable的Cook结果对比

The comparison of Cook results after tracking the ImportTable (left being the default UE Cook results):
追踪了ImportTable的结果对比

As can be seen, the resources Cooked after tracking the ImportTable are consistent with the resources Cooked by UE by default.

Notes

If you have registered some configurable options in the engine, utilizing FSoftObjectPath and marking storage as config on the UObject will imply that the marked resources will be included in the Cook process, which should be avoided. It is advisable to create resource types similar to PrimaryAssetLabel for resource management marking.

Summary

This article has analyzed the engine’s default resource packaging process, transforming UE’s resource analysis from a black box into a deterministic process. By understanding the analysis results, we can obtain a rough list of packaged resources through dependency analysis and implement complete resource packaging by tracking UObject creation during Cook, thereby ensuring consistency with the default Cook resources in UE.

Analyzing UE’s resource packaging process allows us to replace the default Cook process in UE and realize multi-process, even cross-machine Cook task distribution, significantly enhancing UE’s packaging efficiency. In the future, I will implement a MultiCook mechanism for HotPatcher.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:UE Resource Management: Engine Packaging Resource Analysis
Author:LIPENGZHA
Publish Date:2021/12/31 14:33
Word Count:9.8k Words
Link:https://en.imzlp.com/posts/22570/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!