A Flexible and Non-Intrusive Basic Package Splitting Scheme

UE’s default resource management is relatively complex. By default, the resource packaging process is executed based on the configurations of maps, directories, and PrimaryAsset specified in ProjectSetting, as well as a combination of condition checks. Furthermore, UE’s Cook dynamically adds resources to the package based on runtime loading, making the resource packaging process almost a black box.

The blog introduces articles on default resource packaging rules and basic package splitting:

This article presents a new idea, utilizing HotPatcher‘s precise Cook and packaging mechanism to implement a HotChunker Mod that can perform non-intrusive, direct reuse of UE’s default packaging process and clearly split the basic package. This article will specifically introduce the usage and implementation principles.

Issues with Default Packaging

As mentioned earlier, the UE packaging parameters and configurations are complex, and there are some unresolvable issues:

It is impossible to directly estimate which resources will enter the package and the final package size.
The packaging configurations can be scattered in various places, making them difficult to manage.
All Non-uasset resources are in package 0 (meaning they must enter the basic package).
It is impossible to split ShaderLibrary based on package splits (which can be very large).
It is impossible to split AssetRegistry based on package splits.
The ability to precisely control the ignored resources of a certain package rule.
A single set of PakFilter rules shared across all platforms.

From a resource management perspective, it is hoped to have a clear and precise control of resource packaging rules; the default options provided by UE cannot achieve this.

Clean Basic Package

In the article UE Resource Management: Engine Packaging Resource Analysis, I analyzed the resource analysis rules when the engine packages resources.
In addition to resources, there are many Non-uasset files, such as: Slate images, Ini files, ShaderLib/AssetRegistry, and paths configured in the project for DirectoriesToAlwaysStageAsUFS.

A common situation is that when a new project is created, many resources are added, and no packaging rules are configured, executing packaging will package many unnecessary resources.

While this is user-friendly for beginners, it becomes a significant burden when wanting to manage things finely.

Firstly, the first question is: which are the necessary files for engine startup that must be in the basic package?
I summarize the following:

SlateResources, Ini
Global ShaderLib
AssetRegistry
Configured startup maps, along with dependent blueprint GameMode, GameInstance, and other Gameplay framework resources.
Custom joystick resources for mobile terminals.
Basic UI resources that may be used.

During the UE default packaging process, these resources and files will be analyzed and added to the package. However, to pack a clean basic package, we need to isolate the process for these necessary resources and allow the engine to analyze them, while additional non-essential resources are controlled by the business itself.

The second question is: how to exclude non-essential resources?

In the article Engine Packaging Resource Analysis #AssetManager, a condition check in CookOnTheFlyServer is recorded:

If no resources are explicitly specified on the command line and no resources are retrieved from FGameDelegates::Get().GetCookModificationDelegate() or UAssetManager::Get().Modify, then all resources from plugins and the project will be added.

The execution conditions are:

AlwaysCookMaps in DefaultEditor.ini is empty, and AllMaps is empty.
The project settings List of maps to include in a packaged build is empty.
The project settings DirectoriesToAlwaysCook is empty.
FGameDelegates::Get().GetCookModificationDelegate() retrieves no resources.
UAssetManager::Get().ModifyCook retrieves no resources.

This means that not configuring any resources does not make the basic package smaller; it may actually make it larger. To avoid including all resources in the package, we need to feed some resources to CookOnTheFly using the methods above to prevent it from adding unmarked project resources.

The approach I chose is to overload AssetManager and override the ModifyCook function:

void UHotChunkerAssetManager::ModifyCook(TArray<FName>& PackagesToCook, TArray<FName>& PackagesToNeverCook)  
{  
   FName DefaultAsset = TEXT("/Engine/EngineMaterials/DefaultDiffuse");  
   PackagesToCook.Add(DefaultAsset);  
}

Then, specify the AssetManager class as UHotChunkerAssetManager in project settings:

Executing CookCommandlet again shows that this condition is no longer satisfied:

The previous discussion dealt with resource control during the Cook phase. After Cook, it’s essential to control what resources enter the Pak.

In 4.27, all resources under Cooked in the current packaging task are added to the package by default:

If we want to produce a clean package, we need to control which files actually enter package 0, meaning only necessary files should be stored in [PROJECTDIR]/Saved/Cooked.

I added a feature to HotPatcher that allows custom specification of the path where resources are saved during Cook:

This allows for a non-intrusive Cook, thus achieving the separation of the basic package splitting.

Intervening in CookCommandlet

By default, we cannot directly intervene in UE’s packaging process.

UE’s packaging is executed by running UAT to perform BuildCookRun, a standalone program that acts like a task scheduler, launching multiple processes to execute different stages of tasks to orchestrate the entire packaging process and can skip certain stages.

This creates a problem, as it is inconvenient to intervene in this process because UAT is independent of our business code.

However, if modifications to the engine are necessary, that becomes problematic; it is not a universal solution. Modifying the engine every time there is a problem is too simplistic and not my style, so I researched a non-intrusive approach.

From the diagram above, we can see that UAT executes a commandlet for Cook, so while we cannot directly intervene in UAT’s flow, we can use some clever methods to intervene in the CookCommandlet process.

CookCommandlet is a Commandlet executed with:

1	UE4Editor-cmd.exe PROJECT.uproject -run=CookCommandlet

Note, it essentially starts the project through the engine to execute a command line task. This means that when it starts, it will load the plugins within our project.

By leveraging this, we can achieve a non-intrusive intervention in the CookCommandlet process.

Next, we need to determine at what stage we want to do something in the CookCommandlet. For the needs of this article, we hope to invoke HotPatcher to execute the split packaging after Cook completes.

So at what timing is appropriate?

In previous articles, it has been introduced that UE’s Cook dynamically cooks the resources loaded into memory at runtime, meaning we cannot invoke our custom packaging task before Cook completes; otherwise, it would lead to inclusion in the basic package.

We can only do this after Cook finishes, but there is no exposed callback for completion, so we need to consider other solutions.

Because CookCommandlet is also an engine startup, we can use the engine’s exposed CoreDelegates timing to make choices. By reading the code, we find that a suitable timing is OnEnginePreExit, as this phase confirms that Cook is complete, and the engine hasn’t begun exiting yet, maintaining a full engine start state. Actions taken during this phase can utilize all engine functionalities without the risk of some modules being unloaded.

Note that this callback only exists in version 4.25 and later; earlier engine versions do not have it.

First, create a plugin, set up an Editor module, and listen to FCoreDelegates::OnEnginePreExit in StartupModule:

void FHotChunkerEditorModule::StartupModule()  
{  

   #if ENGINE_MAJOR_VERSION > 4 || (ENGINE_MAJOR_VERSION == 4 && ENGINE_MINOR_VERSION > 24)  
       FCoreDelegates::OnEnginePreExit.AddRaw(this,&FHotChunkerEditorModule::OnPreEngineExit_Commandlet);  
    #endif  
}

In the listened callback, we can perform desired actions. However, we need to ensure that this is executed in the CookCommandlet, because the Editor module is executed whenever the engine starts; we simply need it to be effective in the CookCommandlet context:

bool CommandletHelper::IsCookCommandlet()  
{  
   bool bIsCookCommandlet = false;  
  
   if(::IsRunningCommandlet())  
   {     
      FString CommandletName;  
      bool bIsCommandlet = CommandletHelper::GetCommandletArg(TEXT("-run="),CommandletName); 
      if(bIsCommandlet && !CommandletName.IsEmpty())  
      {         
         bIsCookCommandlet = CommandletName.Equals(TEXT("cook"),ESearchCase::IgnoreCase);  
      }   
   }  
   return bIsCookCommandlet;  
}
void FHotChunkerEditorModule::OnPreEngineExit_Commandlet()  
{  
   if(CommandletHelper::IsCookCommandlet())  
   {  
      // do something
   }
}

Through this method, it becomes possible to non-intrusively listen for when Cook is complete and implement custom packaging actions.

Automated Staging

The previous section discussed how to use the Hook Commandlet approach to invoke HotPatcher for packaging after Cook is completed.
After executing CookCommandlet, the packaging status is as follows:

Basic resources have been cooked but not yet packaged into Pak.
Resources split by HotChunker have already been cooked and packaged into Pak, temporarily stored in a staging directory.

By default, CookCommandlet does not package into Pak. The task of packaging the results of CookCommandlet into Pak is performed by UAT invoking UnrealPak.

The Pak we packaged in CookCommandlet does not directly reside in the Saved/StagedBuilds directory because subsequent creation of Pak by UAT will clear the StagedBuilds directory, making copying over before clearing essentially meaningless as it would be deleted.

Therefore, we need to find a slightly later opportunity where our created Pak will not be cleared.

Refer to this diagram:

The relevant code in the engine (Scripts/CopyBuildToStagingDirectory.Automation.cs):

This executes the cleanup prior to UAT invoking UnrealPak, and directly generates the output of UnrealPak into a clean StagedBuilds directory.

Thus, we need to execute our actions after UnrealPak but before the task concludes.

How to do this without modifying the engine?

Here, we need to introduce some basics about loading UE module types.
UE modules can have multiple types usually utilized in daily development like Runtime, Editor, Developer, etc. However, UE also provides a Program type that can allow the loading of some programs when starting.

Using HotChunker’s .uplugin as an example:

{   
   "FriendlyName": "HotChunker",  
   "CreatedBy": "lipengzha",  
   "CreatedByURL": "https://imzlp.com/",  
   "CanContainContent": false,  
   "SupportedPrograms": [ "UnrealPak" ],  
   "Plugins": [  
      {         
         "Name": "HotPatcher",  
         "Enabled": true  
      }  
   ],   
   "Modules": [  
      {         
         "Name": "HotChunker",  
         "Type": "Program",  
         "LoadingPhase": "PostConfigInit",  
         "ProgramAllowList": [ "UnrealPak" ]  
      }
    ]
}

Specifying SupportedPrograms indicates that this plugin can be loaded when a Program starts, and the ProgramAllowList in Modules controls which programs can load that module.

This means we can have UnrealPak load a specific module in our project to perform our intended actions without impacting the engine.

Note, this mechanism is fundamentally provided by the engine to enable non-intrusive extension of compression algorithms, and the way described in this article is a reasonable application.

Of course, this requires passing the project path when starting the Program, which UAT does when invoking UnrealPak.

Within the packaging, the command executed by UAT to invoke UnrealPak is as follows:

1
2
3

E:\UnrealProjects\BlankExample\BlankExample.uproject E:\UnrealProjects\BlankExample\Saved\StagedBuilds\WindowsNoEditor\BlankExample\Content\Paks\BlankExample-WindowsNoEditor.pak -create=D:\UnrealEngine\Source\UE_4.27.2\Engine\Programs\AutomationTool\Saved\Logs\PakList_BlankExample-WindowsNoEditor.txt -cryptokeys
=E:\UnrealProjects\BlankExample\Saved\Cooked\WindowsNoEditor\BlankExample\Metadata\Crypto.json -secondaryOrder=E:\UnrealProjects\BlankExample\Build\WindowsNoEditor\FileOpenOrder\CookerOpenOrder.log -patchpaddingalign=2048 -platform=Windows -compressionformats=Oodle -compressmethod=Kraken -compresslevel=3  -multiprocess -abslog=D:\UnrealEngine\Source\UE_4.27.
2\Engine\Programs\AutomationTool\Saved\Logs\UnrealPak-BlankExample-WindowsNoEditor-2022.10.22-13.58.14.txt -compressionblocksize=256KB

Now we can perform custom actions in the Module’s StartupModule or ShutdownModule:

void FHotChunkerModule::StartupModule()  
{  
   bool bIsRunningProgram = FPlatformProperties::IsProgram();  
   UE_LOG(LogHotChunker,Display,TEXT("HotChunker StartupModule,is Program %s"), bIsRunningProgram ? TEXT("TRUE"):TEXT("FALSE"));
}

Note, it is essential to check at runtime whether the module is started as a Program to distinguish between a normal engine startup and Program startup. Use FPlatformProperties::IsProgram() for this check.

At this point, when triggering packaging, the UnrealPak trigger will start our project Module, which can be seen in the UAT logs when UnrealPak is triggered:

1	LogHotChunker: Display: HotChunker StartupModule,is Program TRUE

Within this Module, we simply need to copy the Pak we packaged in CookCommandlet from the temporary directory to the actual packaging StagedBuilds directory.

General Pak Filtering Rules

In the article UE Hot Update: Splitting Basic Packages, I introduced a method to prevent Android Pak from entering OBB and provided an equivalent implementation for iOS.

However, the downside is that it requires modifying the engine, which is not a non-intrusive complete cross-platform solution.

Based on the package splitting scheme in this article, during the execution of UnrealPak invoking HotChunker, we can perform filtering to avoid entering the basic package.

Only one configuration is needed, and it is applicable across all platforms. It supports wildcards, making it easy to specify ignore rules:

Chunk Configuration

The new splitting approach, much like UE’s PrimaryAssetLabel, allows the creation of a HotPatcherPrimaryLabel resource for configuration. This is consistent with the Chunk configuration in the HotPatcher plugin, allowing for the specification of directories, resources, dependency analysis, non-resource file configurations, and forced ignoring of certain directories, all of which can be flexibly configured.

Additionally, the packaging Chunk’s configuration template can be modified in project settings, completely reusing HotPatcher settings:

This allows for control over the naming rules of the generated Pak, etc.

Editor Support

I added right-click menu support for the HotPatcherPrimaryLabel resource:

This enables direct right-clicking to package the current configuration or to add it to HotPatcher‘s Chunk configuration.

Moreover, it is possible to view the resources included in the current Pak through the reference view:

Preview of Pak within Pak

Taking Android as an example, using this packaging approach will allow a certain Pak to be included within a package:

This provides clearer and more flexible advantages.

Conclusion

This article has introduced the shortcomings of UE’s default package splitting and the non-intrusive splitting of basic packages using HotPatcher methods and implementations. By leveraging many hidden features of the engine, the approach is realized through Commandlet Hook and Module for Program, avoiding modifications to the engine.

The proposed solution has been fully implemented and will be released as a Mod for HotPatcher, requiring only simple configuration and executing the default UE packaging to achieve the complete process.