Z's Blog

Multi-stage automated resource inspection scheme in UE

UE中多阶段的自动化资源检查方案
Posted on In , Views:
Symbols count in article: 9.4k Reading time ≈ 24 mins.

In large projects, the scale of resources is immense, and the production teams involved are very diverse, including scenes, characters, UI, animations, effects, blueprints, data tables, and so on. Consequently, managing the volume and specification of resources becomes difficult to control.

For the established resource specifications, art production personnel may struggle to cover 100% of the scenarios, potentially overlooking details unintentionally. In most cases, issues are discovered after the packages are created, and for existing resources, a significant amount of manpower is required for handling them, making review and repair challenging.

Based on this pain point, I previously developed a resource scanning tool that allows convenient editing of rules to scan resources within the project.

Recently, I have conducted a comprehensive upgrade of the plugin, enhancing its capabilities during editing and automated checks. This article will introduce how to utilize ResScannerUE to perform resource scans during editing, submitting, CI timed or Hook tasks, and Cooking phases, aiming to expose and prompt solutions to problematic resources as early as possible in production, thus avoiding anomalies in package resources.

In terms of specific implementation, many optimizations have been made for scanning speed, transforming the checking process into a nearly unnoticed action, which will be elaborated upon in the article.

Introduction to ResScanner

Two articles have previously discussed its functionality, so I won’t elaborate further but will briefly introduce its features.

ResScannerUE is a resource scanning tool for UE that allows for extremely convenient rule editing, supporting pure configuration and blueprint script rules, with no need to write any C++ code in 99% of cases. For checks on paths and naming, loading resources is not necessary.

It supports four types of rules: path rules, naming rules, property rules, and custom rules, and can extend rules using blueprints or C++. Additionally, a single rule supports multiple conditional combinations and logical operations (AND/OR), providing strong support for complex rules.

For property check rules, based on UE’s reflection and Detail Customization mechanism, it can list all properties based on the selected resource type and construct controls for that property type according to the selected properties, allowing property checks to be completed with mouse clicks:

Deep integration with Git allows scanning based on version management, supporting diff scans between files to be submitted and submission records.

It features comprehensive Commandlet support, enabling integration into CI/CD systems in a configurable and command-line parameter form.

It supports rule scanning during editing, nipping potential errors in the bud.

Editing

Most resource issues are typically caused by the following scenarios:

  1. Temporary resources created in the official asset directory.
  2. Options forgotten during selection or settings.
  3. Incorrect resource specifications.
  4. Submission of resources that should not have been modified.

Such instances are almost unavoidable as the team scale grows. Moreover, for point 4, resources are in binary form and cannot undergo diff comparisons, leading to cases of overwriting; if problems arise, the overwritten resources must be rolled back or recreated, resulting in substantial manpower waste.

Based on this pain point, I added a real-time reminder mechanism to ResScannerUE for editing, scanning resources simultaneously while saving. If non-compliant elements are detected, timely reminders are given:

Additionally, it can check whether the current editor has permission to modify the resource:

The plugin does not prevent saving but provides modification prompts.

In implementation, it listens to UPackage::PackageSavedEvent, and upon saving a resource, it executes this Delegate to determine which resources have been modified:

1
UPackage::PackageSavedEvent.AddRaw(this,&FResScannerEditorModule::PackageSaved);

Checks are performed in the callback:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
void FResScannerEditorModule::PackageSaved(const FString& PacStr,UObject* PackageSaved)
{
  bool bEnable = GetDefault<UResScannerEditorSettings>()->bEnableEditorCheck;
  if(bEnable)
  {
    static UResScannerEditorSettings* ResScannerEditorSettings = GetMutableDefault<UResScannerEditorSettings>();
    ScannerProxy->SetScannerConfig(ResScannerEditorSettings->EditorScannerConfig);

    FString PackageName = LongPackageNameToPackagePath(FPackageName::FilenameToLongPackageName(PacStr));
    FSoftObjectPath ObjectPath(PackageName);
    FAssetData AssetData;
    if(UAssetManager::Get().GetAssetDataForPath(ObjectPath,AssetData))
    {
      const auto& ScanResult = ScannerProxy->ScanAssets(TArray<FAssetData>{AssetData});

      if(ScanResult.HasValidResult())
      {
        FString ScanResultStr = ScanResult.SerializeResult(true);
        UE_LOG(LogResScannerEditor,Warning,TEXT("\n%s"),*ScanResultStr);
        FText DialogText = UKismetTextLibrary::Conv_StringToText(ScanResultStr);
        FText DialogTitle = UKismetTextLibrary::Conv_StringToText(TEXT("ResScanner Message"));
        FMessageDialog::Open(EAppMsgType::Ok, DialogText,&DialogTitle);
      }
    }
  }
}

Since each scan checks only one resource and the saved resource has already been loaded by the engine, the scanning speed is very fast, barely noticeable.

Enabling Method
Open Project Settings - Game - Res Scanner Settings panel to enable bEnableEditorCheck, create a new FScannerMatchRule DataTable, and specify in the rule data table.

Each entry in this DataTable constitutes a separate rule.

You can edit the scanning rules based on your project’s situation, such as scanning only IMPORTANT type rules and supporting black-and-white list specifications.

Submission

In the resource submission phase check, the primary method utilized is the HOOK mechanism provided by version control software, such as Git’s Pre-Commit hook and various hooks provided by Ugit. Essentially, this allows execution of a script during submission that triggers our resource scanning functionality, prohibiting submission when triggers are activated.

This implementation is detailed in my article Automated Resource Checking Practice Based on ResScannerUE.

CI/CD

Resource scanning can be integrated into the CI/CD system for timed or Hook-triggered scans:

Using the plugin’s Commandlet mechanism, this can be achieved through exported configuration files and command-line parameters.

1
UE4Editor.exe Project.uproject -run="ResScanner" -config="Full_Scan_Config.json" -gitChecker.bGitCheck=true -gitchecker.beginCommitHash=HEAD~ -gitchecker.endCommitHash=HEAD

After scanning, group notifications, along with @mentions of relevant submitters, can be sent using corporate WeChat bots and other methods.

I have provided a complete implementation script; details can be found here: Corporate WeChat Notifications

During Packaging

Not all resources in the engine will be fully packed; if you only want to analyze the rule checks within the currently packed resources, the plugin also provides support.

As long as it is enabled, it will automatically execute during the Cook phase and generate a detection report after cooking, which is by default stored in Saved/ResScanner/Cooking.txt.

The entire implementation is non-intrusive, requiring integration of the plugin without any changes to the engine.

Enabling Method

Enable bEnableCookingCheck in Project Settings - Game - ResScanner Settings, and configure the rules for scanning during Cook:

Scanning Optimization

Resource scanning essentially involves launching the UE4Editor-cmd process to execute Commandlets, which is somewhat time-sensitive as it requires the complete engine to be initialized.

Therefore, some optimizations need to be made regarding Commandlet startup to avoid prolonged engine startup leading to flow stalling.

First, analyze the startup timing of each module during engine launch, specifically the time taken by each module’s StartupModule. The engine does not have a default profiling tag to analyze all Modules, but you can add profiling tags in FModuleDescriptor::LoadModulesForPhase and FModuleManager::LoadModule for checks:

Runtime\Projects\Private\ModuleDescriptor.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
void FModuleDescriptor::LoadModulesForPhase(ELoadingPhase::Type LoadingPhase, const TArray<FModuleDescriptor>& Modules, TMap<FName, EModuleLoadResult>& ModuleLoadErrors)
{
    FScopedSlowTask SlowTask(Modules.Num());
    for (int Idx = 0; Idx < Modules.Num(); Idx++)
    {
        SlowTask.EnterProgressFrame(1);
        const FModuleDescriptor& Descriptor = Modules[Idx];

        // Don't need to do anything if this module is already loaded
        if (!FModuleManager::Get().IsModuleLoaded(Descriptor.Name))
        {
            if (LoadingPhase == Descriptor.LoadingPhase && Descriptor.IsLoadedInCurrentConfiguration())
            {
                // @todo plugin: DLL search problems.  Plugins that statically depend on other modules within this plugin may not be found?  Need to test this.

                // NOTE: Loading this module may cause other modules to become loaded, both in the engine or game, or other modules 
                //       that are part of this project or plugin.  That's totally fine.
                FScopedNamedEventStatic LoadModuleEvent(FColor::Red,*Descriptor.Name.ToString());
                EModuleLoadResult FailureReason;
                IModuleInterface* ModuleInterface = FModuleManager::Get().LoadModuleWithFailureReason(Descriptor.Name, FailureReason);
                if (ModuleInterface == nullptr)
                {
                    // The module failed to load. Note this in the ModuleLoadErrors list.
                    ModuleLoadErrors.Add(Descriptor.Name, FailureReason);
                }
            }
        }
    }
}

FModuleManager::LoadModule:

Runtime\Core\Private\Modules\ModuleManager.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
IModuleInterface* FModuleManager::LoadModule( const FName InModuleName )
{
    FScopedNamedEventStatic LoadModuleEvent(FColor::Red,*InModuleName.ToString());
    // We allow an already loaded module to be returned in other threads to simplify
    // parallel processing scenarios but they must have been loaded from the main thread beforehand.
    if(!IsInGameThread())
    {
        return GetModule(InModuleName);
    }

    EModuleLoadResult FailureReason;
    IModuleInterface* Result = LoadModuleWithFailureReason(InModuleName, FailureReason );

    // This should return a valid pointer only if and only if the module is loaded
    checkSlow((Result != nullptr) == IsModuleLoaded(InModuleName));

    return Result;
}

AkAudio

If the project inherits Wwise audio functionality, its AkAudio module startup will be very time-consuming (depending on the project’s resource volume). After analysis, it was found that it scans all files under the project’s Content folder and scans AssetRegistry data during StartupModule, which is extremely time-consuming, taking several tens of seconds to simply scan files:

AkAudioModule.cpp
1
2
3
4
5
6
7
#if WITH_EDITOR
  TArray<FString> paths;
  IFileManager::Get().FindFilesRecursive(paths, *FPaths::ProjectContentDir(), TEXT("InitBank.uasset"), true, false);
  
  FAssetRegistryModule& AssetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>(TEXT("AssetRegistry"));
  AssetRegistryModule.Get().ScanFilesSynchronous(paths, false);
#endif

These two operations are exceedingly time-consuming, but they are unnecessary for Commandlets and can be shielded by checking ::IsRunningCommandlet.

Note: In the EBP mode of Wwise, it’s essential to verify the existence of InitBank in the AssetRegistry; else, another one will be created, prompting a window.
The solution to this issue is to generate Registry data for the InitBank resource while ignoring the overall scan.

1
2
3
4
5
FString PackageFilename;  
if(FPackageName::FindPackageFileWithoutExtension(FPackageName::LongPackageNameToFilename(TEXT("/Game/WwiseAudio/InitBank")), PackageFilename))  
{ 
    AssetRegistryModule.Get().ScanModifiedAssetFiles(TArray<FString>{PackageFilename});  
}

LiveCoding

If the project has LiveCoding enabled, its startup time can be considerable:

However, Commandlets do not need support related to LiveCoding, and directly disabling it could impact daily development.

Thus, I implemented a method that dynamically disables LiveCoding while executing Commandlets, so even if LiveCoding is always enabled in Editor Settings, it can be turned off during the Commandlet execution.

The approach is to modify the configuration in GEditorPreProjectIni during the StartupModule of a higher-priority module (Default):

1
2
3
4
if(IsRunningCommandlet())
{
  GConfig->SetBool(TEXT("/Script/LiveCoding.LiveCodingSettings"),TEXT("bEnabled"),false,GEditorPerProjectIni);
}

This way, when the LiveCoding module starts, it will recognize that it is disabled and will avoid executing those time-consuming tasks.

Asset Registry

During engine startup, AssetManager scans PrimaryAsset. If not cached, rebuilding AssetRegistry data from resources is also a time-consuming process.


For resource scanning, most of the time (especially when executed locally), complete AssetRegistry data is unnecessary. Scanning can be restricted by closing SearchAllAssets and ScanPrimaryAssetTypesFromConfig, generating corresponding AssetRegistry data on-demand during resource checks.

No engine modification is needed; you just need to inherit a class of UAssetManager and override the ScanPrimaryAssetTypesFromConfig interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// .h
UCLASS()
class HOTPATCHERRUNTIME_API UHotPatcherAssetManager : public UAssetManager
{
	GENERATED_BODY()
public:
	virtual void ScanPrimaryAssetTypesFromConfig() override;
};
// .cpp
void UHotPatcherAssetManager::ScanPrimaryAssetTypesFromConfig()
{
	SCOPED_NAMED_EVENT_TEXT("ScanPrimaryAssetTypesFromConfig",FColor::Red);
	static const bool bNoScanPrimaryAsset = FParse::Param(FCommandLine::Get(), TEXT("NoScanPrimaryAsset"));
	if(bNoScanPrimaryAsset)
	{
		UE_LOG(LogHotPatcher,Display,TEXT("Skip ScanPrimaryAssetTypesFromConfig"));
		return;
	}
	Super::ScanPrimaryAssetTypesFromConfig();
}

When executing the Commandlet, the -NoScanPrimaryAsset parameter can be used to disable AssetRegistry scanning. Moreover, avoid executing SearchAllAssets within Commandlet code as it is also time-consuming.

However, during runtime, if accessing AssetRegistry data is necessary, targeted scans for the required package’s AssetRegistry data are already supported within the plugin.

Effect

After optimization, when both the engine and client, along with content, are placed on an SSD, the overall engine startup duration can be reduced to approximately 20 seconds, and the resource scanning process can be under 20 seconds, making the submission experience nearly unobtrusive.

In an HDD scenario, engine startup and scanning durations can be more than double; this is due to IO bottlenecks, emphasizing the importance of having all resources on solid-state storage.

Update Log

2023.03.30 Update

  • Supports progressive scanning.
  • Optimized the scanning implementation during cooking, allowing individual resource scanning during the cook phase to reduce loading times.
  • Enhanced dependencies in reference relationship scanning efficiency, and added derivative class caching.
    Before Optimization
    After Optimization
  • Refined the scanning process to avoid executing unnecessary rules.
  • Optimized the serialization phase of scanning results, supporting non-CookOnTheFly modes.
  • Improved methods of tracking warning/error-level logs during cooking and deduplicated them.

Conclusion

This article discussed how to utilize ResScannerUE to implement checks at various resource submission stages, covering the vast majority of scenarios in practical development and aiming to expose and resolve errors proactively.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:Multi-stage automated resource inspection scheme in UE
Author:LIPENGZHA
Publish Date:2022/08/23 11:30
World Count:9.4k Words
Link:https://en.imzlp.com/posts/22655/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!