Z's Blog

UE Resource Scanning and inspection tool ResScannerUE

UE资源合规检查工具ResScannerUE
Posted on In , Views:
Symbols count in article: 12k Reading time ≈ 29 mins.

In game project development, due to the large amount of resources involved and the broad range of personnel, it’s relatively difficult to consciously unify resource standards. If issues arise with the resources, manually checking them requires a significant amount of manpower. This highlights the risks and pain points in resource management.

Based on this demand, I developed a resource scanning standard tool under the editor, ResScannerUE, which allows for easy configuration of rules and automation. This can be executed by artists before submitting resources or on a regular basis to check whether the resources in the project comply, thus avoiding issues that may only be discovered after packaging.

This article introduces the usage methods, operational mechanisms, ways to extend custom rules, and future optimization arrangements for the ResScannerUE plugin.

Plugin Introduction

ResScannerUE is a resource scanning tool implemented based on the AssetRegistry and reflection mechanism, with the following advantages:

  1. Name and path rule checks do not require loading of resources.
  2. Name and path rules provide support for wildcards.
  3. Options in resources can be checked by providing property names and values.
  4. Direct selection support for property names and values eliminates the need for input.
  5. Very convenient to customize and extend check rules (implementing with Blueprints or C++).
  6. Supports any resource type, most requirements can be implemented with Zero-Code.
  7. Supports import/export of configurations.
  8. Supports automation via Commandlet.
  9. Supports version comparison of Git commits and comparison of pending files.
  10. Supports resource submission permission checks.

Naming, path, and properties can strictly match multiple, or match one from multiple rules, enabling expressions like ((expression) && (expression || expression), such as matching patterns that start with T_ and end with _x or _d).

Interface preview (maintaining consistency with the HotPatcher style):

Plugin Parameters

FScannerConfig

  • ConfigName: The name of the current configuration.
  • bByGlobalScanFilters: Enables the global scanning configuration used by all rules; when enabled, resource configurations in each rule do not take effect.
  • bBlockRuleFilter: Shields resources configured in each rule, using only global resources.
  • GlobalScanFilters: Depends on bByGlobalScanFilters, the global scanning configuration used by all rules (ignores scanning configurations specified in each rule).
  • GlobalIgnoreFilters: Resources ignored in all rules (paths, specific resources).
  • GitChecker: Enables GIT version checking.
  • bRuleWhiteList: Enables rule whitelist, checking only specific rules.
  • RuleWhileListIDs: Whitelist rule ID list, Note: starts from 0, and IDs in RuleTable should be -1.
  • bUseRulesTable: Whether to use a rules data table (recommended).
  • ImportRulesTable: Specifies a Datatable of FScannerMatchRule.
  • ScannerRules: Array of FScannerMatchRule rules for specifying resource check rules.
  • bSaveConfig: Whether to save the configuration file.
  • bSaveResult: Whether to save the results of this execution.
  • SavePath: Storage path for Config and Result.

FGitChecker

Able to extract files in Git commits for checking:

  • bGitCheck: Enables Git repository scanning.
  • bRecordCommiter: Records the committer.
  • RepoDir: GIT repository address.
  • bDiffCommit: Git commit record comparison (Begin/End Commit HASH).
  • BeginCommitHash: The starting Git Commit HASH for checking.
  • EndCommitHash: The ending Git Commit HASH for checking.
  • bUncommitFiles: Checks pending files.

FScannerMatchRule

Each rule supports the following parameters:

  • RuleName: Rule name.
  • RuleDescribe: Rule description information.
  • bEnableRule: Whether to enable the current rule.
  • Priority: The priority of this rule.
  • bGlobalAssetMustMatchFilter: Must match the Filter directory in the rule (global resource); for example, if different specifications for textures are set in different directories, only resources in certain directories should use this rule for scanning.
  • ScanFilters: Array of FDirectoryPath, specifies scanning resource paths.
  • ScanAssetTypes: Array of UClass, specifies resource types to be scanned; it is recommended to specify only one type for each rule.
  • NameMatchRules: Array of FNameMatchRule for specifying naming matching rules.
  • PathMatchRules: Array of FPathMatchRule for specifying resource path matching rules.
  • PropertyMatchRules: Array of FPropertyMatchRule for matching properties in resources.
  • CustomRules: Array of TSubclassOf<UOperatorBase> for custom matching rules; can specify C++/Blueprint classes.
  • IgnoreFilters: The ignore list for this rule (paths, specific resources).
  • bEnablePostProcessor: Whether to enable post-processing of scan results.
  • PostProcessor: Array of TArray<TSubclassOf<UScannnerPostProcessorBase>>, relies on bEnablePostProcessor; can specify post-processing of the current rule’s scan results, such as automatically modifying properties, auto-renaming, etc.

FNameMatchRule

Each NameMatchRule contains multiple NameRule:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
USTRUCT(BlueprintType)
struct FNameRule
{
  GENERATED_USTRUCT_BODY()
public:
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  ENameMatchMode MatchMode;
  // Matching rule; whether required or optional; Necessary means all rules must be matched, Optional means only one in the rules must be matched.
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  EMatchLogic MatchLogic;
  // UPROPERTY(EditAnywhere,BlueprintReadWrite,meta=(EditCondition="MatchLogic == EMatchLogic::Optional"))
  int32 OptionalRuleMatchNum = 1;
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  TArray<FString> Rules;
  
};

USTRUCT(BlueprintType)
struct FNameMatchRule
{
  GENERATED_USTRUCT_BODY()
public:
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  TArray<FNameRule> Rules;
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  bool bReverseCheck;
};

FPathMatchRule

Each PathMatchRule contains multiple PathRule:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
USTRUCT(BlueprintType)
struct FPathRule
{
  GENERATED_USTRUCT_BODY()
public:
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  EPathMatchMode MatchMode;
  // Matching rule; whether required or optional; Necessary means all rules must be matched, Optional means only one in the rules must be matched.
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  EMatchLogic MatchLogic;
  // UPROPERTY(EditAnywhere,BlueprintReadWrite,meta=(EditCondition="MatchLogic == EMatchLogic::Optional"))
  int32 OptionalRuleMatchNum = 1;
  UPROPERTY(EditAnywhere,BlueprintReadWrite,meta = (RelativeToGameContentDir, LongPackageName))
  TArray<FString> Rules;
};

USTRUCT(BlueprintType)
struct FPathMatchRule
{
  GENERATED_USTRUCT_BODY()
public:
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  TArray<FPathRule> Rules;
  UPROPERTY(EditAnywhere,BlueprintReadWrite)
  bool bReverseCheck;
};

FPropertyMatchRule

Each property matching rule contains multiple property rules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
USTRUCT(BlueprintType)
struct FPropertyRule
{
	GENERATED_USTRUCT_BODY()
public:
	// Matching rule; whether required or optional; Necessary means all rules must be matched, Optional means only one in the rules must be matched.
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	EMatchLogic MatchLogic;
	// UPROPERTY(EditAnywhere,BlueprintReadWrite,meta=(EditCondition="MatchLogic == EMatchLogic::Optional"))
	int32 OptionalRuleMatchNum = 1;
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	TArray<FPropertyMatchMapping> Rules;
	
};
USTRUCT(BlueprintType)
struct FPropertyMatchRule
{
	GENERATED_USTRUCT_BODY()
public:
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	TArray<FPropertyRule> Rules;
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
    bool bReverseCheck;
};

Matching Rules

I provide four default matching methods for ResScannerUE: naming matching, path matching, property matching, submission permission matching, and custom matching, which can meet most resource scanning requirements.

For each matching method, multiple constraints can also be added. Taking the texture naming rule as an example:

  1. Textures must start with T_*.
  2. Based on the application type of the texture, it should end with _c or _d.

Essentially, two constraints composed of and/or operator need to be fulfilled: (T_) && (_x || _d). In the implementation of ResScannerUE, each NameRule is an expression that determines if the internal logic is and/or by MatchLogic.

Each NameRule has an attribute of EMatchLogic. Necessary must strictly match all Rules (starting with T_), whereas Optional means only one of the patterns listed in the Rule needs to match (_x/_d). Path and property matching also supports setting the MatchLogic mode similarly.

Naming Matching

Setting naming rules for different types of resources is one of the most common standards, such as textures usually starting with T_.

In a resource detection rule, multiple path matching rules can also be added, with each matching rule having three modes:

  1. StartWith
  2. EndWith
  3. Wildcard

Additionally, bReverseCheck is provided to negate the results of all matching rules for reverse matching.

Taking the naming standard for textures as an example:

It includes a wildcard check of T_*, and enabling bReverseCheck will record resources that do not match this rule:

result.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "matchedAssets": [
    {
      "ruleName": "Texture2D_Name_Rule",
      "ruleDescribe": "texture name need startwith \"T_\"",
      "ruleId": 0,
      "assetPackageNames": [
        "/Game/Mannequin/Character/Textures/UE4_LOGO_CARD",
        "/Game/Mannequin/Character/Textures/UE4_Mannequin__normals",
        "/Game/Mannequin/Character/Textures/UE4_Mannequin_MAT_MASKA",
        "/Game/Mannequin/Character/Textures/UE4Man_Logo_N",
        "/Game/Texture/20210914162516"
      ]
    }
  ]
}

Path Matching

Checking whether a certain type of resource is stored according to a specified path is also one of the most common rules. It similarly supports multiple path specifications, and each specification default supports two modes:

  1. WithIn
  2. Wildcard

It can check whether the resource is located within a certain directory or whether the path meets the wildcard criteria. Taking textures again as an example, one could check whether all textures are stored in directories named */Textures/*:

It includes a wildcard check for */Textures/* and enabling bReverseCheck will record resources that do not match this rule:

result.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "matchedAssets": [
    {
      "ruleName": "Texture2D_Path_Rule",
      "ruleDescribe": "Texture need within */Textures/* folder.",
      "ruleId": 1,
      "assetPackageNames": [
        "/Game/Mannequin/Character/Materials/MaterialLayers/T_ML_Rubber_Blue_01_D",
        "/Game/Mannequin/Character/Materials/MaterialLayers/T_ML_Rubber_Blue_01_N",
        "/Game/Mannequin/Character/Materials/MaterialLayers/T_ML_Aluminum01",
        "/Game/Mannequin/Character/Materials/MaterialLayers/T_ML_Aluminum01_N",
        "/Game/Texture/20210914162516",
        "/Game/Texture/T_20210914162516"
      ]
    }
  ]
}

Property Matching

Checking whether a certain option is enabled within a resource or checking the value of a specific property is a key aspect of resource compliance checking. Similarly, multiple rules can be added for property attribute detection.

I have implemented a mechanism for property matching based on reflection, wherein any resource type can be checked as long as it has reflection-marked properties. Property matching can be done through property names and detection values, with two modes for value checking:

  1. Equal
  2. NotEqual

For example, checking if CompressionSettings is set to TC_Default and CompressionQuality is set to either TCQ_Highnest or TCQ_High for textures. I have previously described how the MatchLogic array can be used to handle the logical processing. This is also important for property detection.

Successfully matching resources can be retrieved:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "matchedAssets": [
    {
      "ruleName": "Texture_CompressSetting_Rule",
      "ruleDescribe": "Texture2D need Set Compress",
      "ruleId": 2,
      "assetPackageNames": [
        "/Game/Texture/20210914162516"
      ]
    }
  ]
}

Note: Property matching is based on the reflection name of the property; in UE there is DisplayName, which may not match the actual reflection name shown in the editor, so care should be taken. (The plugin now supports property selection)

Property matching allows for custom reflection-based property panels, capable of automatically listing all reflected properties after selecting UClass, and automatically creating property name editor types in the editor based on the selected property name, which is very amazing:

Submission Permission Matching

Supports checking modification permissions of resource files in the repository, allowing only specific personnel to modify certain types of resources. For example, certain master materials in the project may only be modified by the TA team.

There are two modes of checks:

  1. Based on the commit information in Git.
  2. Based on machine username.

When based on Git, pending files can be retrieved, along with the most recent submitter of the file:

When based on machine usernames, the current machine’s username is retrieved, and a StartsWith check is performed since some machine names may include suffixes like lipengzha-PC*.

Editor displays alerts on checks:

Custom Rules

While I have introduced several default matching rules, for some special detection needs, I offer a convenient extension method where subclasses can inherit from UOperatorBase and override the Match function to implement custom detection needs.

1
2
3
4
5
6
7
8
UCLASS(Blueprintable,BlueprintType)
class UOperatorBase : public UObject
{
  GENERATED_BODY()
public:
  UFUNCTION(BlueprintImplementableEvent,BlueprintCallable)
  bool Match(UObject* Object,const FString& AssetType);
};

This will pass the current resource’s UObject and the AssetType string, allowing for customized detection based on this data.

In the Rule’s CustomRules, specify the class to be used (TSubclassOf<UOperatorBase>):

Custom Rules

Alternatively, one can also create Blueprint classes by inheriting UOperatorBase, making it even easier to implement overrides:

When executing a scan, the specified class’s CDO will be obtained, and the Match function will be called to get the matching results.

For example, detecting settings of GameMode on a map can be achieved through a custom rule:

Another example, alerting when resources reference the NeverCook directory to avoid issues caused by missing dependencies during packaging:

Post-Processing After Matching

The plugin provides support for performing post-processing on matching rules, such as automated setting of non-compliant resource properties or auto-renaming needs. It is essential to be able to access matching results.

I provide an interface in each FScannerMatchRule for specifying a UScannnerPostProcessorBase class. If one intends to process resources that match the rules, they can inherit the UScannnerPostProcessorBase class and implement the Processor function. This will pass the scan results to it after the current rule’s scan is complete.

1
2
3
4
5
6
7
8
9
10
11
for(const auto& PostProcessorClass:Rule.PostProcessors)
{
  if(IsValid(PostProcessorClass))
  {
    UScannnerPostProcessorBase* PostProcessorIns = Cast<UScannnerPostProcessorBase>(PostProcessorClass->GetDefaultObject());
    if(PostProcessorIns)
    {
      PostProcessorIns->Processor(RuleMatchedInfo,Rule.ScanAssetType->GetName());
    }
  }
}

Final Scan Results

Once all rule scans are completed, the results are structured in an FMatchedResult, containing a FRuleMatchedInfo for recording the list of resources matched by each rule:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Rule Matched info
USTRUCT(BlueprintType)
struct FRuleMatchedInfo
{
	GENERATED_USTRUCT_BODY()
public:
	FRuleMatchedInfo():RuleName(TEXT("")),RuleDescribe(TEXT("")),RuleID(-1){}
	
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	FString RuleName;
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	FString RuleDescribe;
	// The index of this rule in the configuration array
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	int32 RuleID;
	UPROPERTY(EditAnywhere,BlueprintReadWrite, transient)
	TArray<FAssetData> Assets;
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	TArray<FString> AssetPackageNames;
};
// final result
USTRUCT(BlueprintType)
struct FMatchedResult
{
	GENERATED_USTRUCT_BODY()
public:
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	TArray<FRuleMatchedInfo> MatchedAssets;
};

RuleName, RuleDescribe, and RuleID provide a convenient way to determine which rules the resources matched against.

Ultimately, this structure will be serialized into JSON output. If you want to obtain the original results or perform some special processing, you can call GetScanResult() in UResScannerProxy.

Automated Checks

ResScannerUE offers methods for executing via Commandlet, initiated by specifying the configuration file:

1
UE4Editor-cmd.exe PROJECT_NAME.uproject -run=ResScanner -config="res_scanner.json"

One can add the -wait parameter to wait for input after execution completes, allowing for previewing results.

Utilizing another previously open-sourced tool, UELauncher, can facilitate configuration:

For practical automated scanning implementations based on ResScannerUE, refer to my other article: Automated Practices for Resource Checks Based on ResScannerUE

Update Log

2022.08.02 v25

ResScannerUE v25 Released!

  • Support for checking Git pending files.
  • Support for scanning TextureCube specifications.
  • Support for scanning GameMode settings in maps.
  • Deepened Git adaptation, supporting checks that certain resources can only be modified by specific users.
  • Support for retrieving the modifier of currently changing resources in rules.
  • Support for getting user local git information (username/email) in the plugin.
  • Support for passing git username via commandlet.
  • Support for checking pre-commit hooks.

For the following case requirement: only specific users can submit Material resources, while others are denied submission.
A rule filtering Material resources was created and a custom rule specified:

In the scanning configuration, enable global resources and Git repository scanning configurations:

In the custom rule blueprint, obtain the status of locally modified files and the committer:

Execute the scan:

Additionally, it can be integrated into Git GUI tools or pre-commit hooks to achieve fully automated checks:

2021.13.15 v24

  • Fixed failure to retrieve commit information for maps from Git.
  • Fixed crash issues on Mac.
  • Disabled shader compilation during commandlet scanning.

Optimization Plans

Some thoughts for future optimizations; I will gradually implement these when time permits.

  • Implement property name selection instead of input.
  • Create property based on specific types in the Detail panel.
  • Multi-threaded rule detection for parallel processing.
The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:UE Resource Scanning and inspection tool ResScannerUE
Author:LIPENGZHA
Publish Date:2021/09/15 15:29
Word Count:12k Words
Link:https://en.imzlp.com/posts/11750/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!