Automated Resource Scanning and Inspection with ResScannerUE

For the resource inspection requirements in the project, it needs to be simple and convenient to configure, automate execution, and be able to promptly locate related personnel. Based on this requirement, I have open-sourced a resource compliance scanning tool ResScannerUE, with configuration documentation: UE Resource Compliance Inspection Tool ResScannerUE.
This article will introduce how to achieve automated resource scanning through this tool, provide support for incremental detection combined with Git, use Commandlet to automatically trigger and execute detection rules for Content content changes on CI platforms, and be able to locate the most recent committers of problematic resources, achieving precise positioning and real-time scanning report sending to corporate WeChat, reminding relevant personnel to take action.
Additionally, I also provide a Git-based Pre-Commit Hook implementation, allowing detection of non-compliant resources before submitting and prohibiting the submission, thus avoiding the contamination of remote repositories with problematic resources. The overall solution has been meticulously designed and extensively optimized for experience and enhanced automation support, making it very convenient to configure and integrate, capable of meeting various resource scanning needs.

In the plugin, it is possible to configure paths and resource lists for each rule, and it also provides global resource paths and resource lists used for all rules, supporting the suppression of configurations in each rule.

By controlling global resources and suppressing resource configurations in single rules, it can precisely control which resources undergo all rules checks, which is the basis for achieving automated resource scanning.

To implement automated resource scanning, it can be divided into two parts based on different needs:

1. Full scan
2. Incremental scan

The full scan goes without saying; you simply need to specify the global resource /Game and other Root directories in the plugin, suppress the resources configured in the rules, and export a configuration. For incremental scans, I provide two solutions in the plugin: detection based on a file list and detection based on Git version comparison.

File List Detection

If you want to dynamically specify which resources to check rather than pre-specifying them in the configuration, I have provided functionality in the ResScanner commandlet to specify a file list. You can add -filecheck and -filelist to the execution parameters to specify the file list, separating files with a comma.

UE4Editor.exe Project.uproject -run="ResScanner" -config="ScannerConfig.json" -filecheck -filelist="Asset/A.uasset,Asset/B.uasset"

This method requires passing all files to be checked externally.

Git Version Comparison Detection

Typically, project repositories are divided into two parts:

1. Code repository: basic project structure, code, configurations, etc.
2. Resource repository: Content

This is used to isolate the working environments of art and programming from each other. For most resource management situations, version control tools like Git are used for management. Therefore, it is necessary to extract the file list from each commit in the resource repository managed by Git and pass it to ResScanner for specific resource checks.

The original Git comparison between two commits can be done using the following command:

git diff --name-only HEAD~ HEAD

This can be used to get the file changes between the last two commits, thus obtaining the original list of changed files.

Usually, this part of the logic would be implemented through scripts such as Python, but since this demand is relatively general, I provide support directly in ResScannerUE. When we want to perform rule checks using Git versions, we can enable the provided GitChecker through the plugin.

Git’s version comparison and committer retrieval are based on a tool I previously open-sourced: GitControllerUE, which is a plugin for UE capable of obtaining Git repository version information.

Based on it, ResScannerUE can automatically scan the most recent commit record changes in files and check for non-compliance in submissions. By default, it checks the most recent commit, but if you want to specify more commits, you only need to modify the hash value of the starting commit in the configuration.

The following configuration example scans the files of the latest Git commit for texture naming checks:

{
	"configName": "Full Resource Scan",
	"bByGlobalScanFilters": true,
	"bBlockRuleFilter": true,
	"globalScanFilters":
	{
		"filters": [],
		"assets": []
	},
	"globalIgnoreFilters": [],
	"gitChecker":
	{
		"bGitCheck": true,
		"bRecordCommiter": true,
		"bDiffCommit": true,
		"repoDir":
		{
			"path": "[PROJECT_CONTENT_DIR]"
		},
		"beginCommitHash": "HEAD~",
		"endCommitHash": "HEAD",
		"bUncommitFiles": false
	},
	"bUseRulesTable": false,
	"importRulesTable": "",
	"scannerRules": [
		{
			"ruleName": "Texture Naming Norms",
			"ruleDescribe": "Textures should start with T_",
			"bEnableRule": true,
			"priority": "GENERAL",
			"scanFilters": [],
			"scanAssetType": "Class'/Script/Engine.Texture2D'",
			"recursiveClasses": true,
			"nameMatchRules":
			{
				"rules": [
					{
						"matchMode": "StartWith",
						"matchLogic": "Necessary",
						"rules": [
							{
								"ruleText": "T_",
								"bReverseCheck": false
							}
						]
					}
				],
				"bReverseCheck": true
			},
			"pathMatchRules":
			{
				"rules": [],
				"bReverseCheck": false
			},
			"propertyMatchRules":
			{
				"matchRules": [],
				"bReverseCheck": false
			},
			"customRules": [],
			"ignoreFilters":
			{
				"filters": [],
				"assets": []
			},
			"bEnablePostProcessor": false,
			"postProcessors": []
		}
	],
	"bSaveConfig": true,
	"bSaveResult": true,
	"savePath":
	{
		"path": "[PROJECT_SAVED_DIR]/ResScanner"
	},
	"bStandaloneMode": false,
	"additionalExecCommand": ""
}

You just need to specify this config when executing the Commandlet:

UE4Editor.exe Project.uproject -run="ResScanner" -config="Full_Scan_Config.json"

Additionally, in the GitChecker configuration, there is an option to record the submitter of the files. Here, I made a trick: in the storage structure of the scanning results, there are two arrays:

USTRUCT(BlueprintType)
struct FFileCommiter
{
	GENERATED_USTRUCT_BODY()
public:
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	FString File;
	UPROPERTY(EditAnywhere,BlueprintReadWrite)
	FString Commiter;
}
USTRUCT(BlueprintType)
struct FRuleMatchedInfo
{
    GENERATED_USTRUCT_BODY()
public:
    // ...
    UPROPERTY(EditAnywhere,BlueprintReadWrite)
    TArray<FString> AssetPackageNames;
    UPROPERTY(EditAnywhere,BlueprintReadWrite)
    TArray<FFileCommiter> AssetsCommiter;
};

The scanning result will be serialized into JSON. By default, both arrays will be serialized, but this is not our requirement. When bRecordCommiter is off, only the AssetPackageNames of the scanned resources will be serialized; conversely, AssetsCommiter will be serialized. We can control the serialization based on whether FProperty has the CPF_Transient FLAG. If this FLAG is included, the Property will not be serialized.

Thus, before each serialization of the scanning result, I will perform a marking action, deciding which property to add CPF_Transient based on the configuration:

// set CPF_Transient to AssetPackageNames or AssetsCommiter
static void SetSerializeTransient(bool bCommiter)
{
  FString NotSerializeName = bCommiter ? TEXT("AssetPackageNames") : TEXT("AssetsCommiter");
  for(TFieldIterator<FProperty> PropertyIter(FRuleMatchedInfo::StaticStruct());PropertyIter;++PropertyIter)
  {
    FProperty* PropertyIns = *PropertyIter;
    if(NotSerializeName.Equals(*PropertyIns->GetName()))
    {
      PropertyIns->SetPropertyFlags(CPF_Transient);
    }
  }
}
// clear CPF_Transient
static void ResetTransient()
{
  TArray<FString> NotSerializeNames = {TEXT("AssetsCommiter"),TEXT("AssetPackageNames")};
  for(TFieldIterator<FProperty> PropertyIter(FRuleMatchedInfo::StaticStruct());PropertyIter;++PropertyIter)
  {
    FProperty* PropertyIns = *PropertyIter;
    if(NotSerializeNames.Contains(*PropertyIns->GetName()) && PropertyIns->HasAnyPropertyFlags(CPF_Transient))
    {
    PropertyIns->ClearPropertyFlags(CPF_Transient);
    }
  }
}

Call these two functions before each serialization:

FRuleMatchedInfo::ResetTransient();
FRuleMatchedInfo::SetSerializeTransient(GetScannerConfig()->GitChecker.bGitCheck && GetScannerConfig()->GitChecker.bRecordCommiter);

Thus achieving on-demand serialization effect:

Detection of Uncommitted Files

The previous section introduced detection of resource changes between two committed versions. If the resources are not committed, another mechanism needs to be employed.

Uncommitted files refer to those files with local modifications that have not been committed:

$ git status
On branch master
Your branch is behind 'origin/master' by 49 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   Assets/Scene/BaseMaterials/4X4_MRA.uasset

Based on the detection of uncommitted files, it is possible to perform resource scanning before submitting and prohibit non-compliant submissions. This will be detailed in the following implementation of the pre-commit hook.

In the plugin, there are options in the GitChecker configuration:

"gitChecker":
{
	"bGitCheck": true,
	"bRecordCommiter": true,
	"bDiffCommit": true,
	"repoDir":
	{
		"path": "[PROJECT_CONTENT_DIR]"
	},
	"beginCommitHash": "HEAD~",
	"endCommitHash": "HEAD",
	"bUncommitFiles": false
}

There are two modes for Git detection:

• bDiffCommit: comparison of resource scans based on the two Commit HASH versions
• bUncommitFiles: scanning uncommitted resources

Both modes can be enabled as needed; typically, scheduled scans will enable bDiffCommit, while pre-submission checks will use bUncommitFiles.

Commandlet Parameter Replacement

To further facilitate executing scan tasks through Commandlet, I have also added parameter replacement functionality to the commandlet of ResScannerUE. You can dynamically replace options in the configuration file while specifying it through command line parameters, achieving more powerful automation capabilities.

For example, you can disable global resource configurations or choose whether to store scan results via the command line:

UE4Editor.exe Project.uproject -run="ResScanner" -config="Full_Scan_Config.json" -bByGlobalScanFilters=false -savePath.path="D:\"

And you can specify the starting commit HASH for Git version scans:

UE4Editor.exe Project.uproject -run="ResScanner" -config="Full_Scan_Config.json" -gitChecker.bGitCheck=true -gitchecker.beginCommitHash=HEAD~ -gitchecker.endCommitHash=HEAD

Properties in the plugin can be specified in the manner of -ProjectName=Value. This way, it achieves the requirement of providing a basic configuration, while dynamically controlling the behavior of parameters in the configuration via the command line, exposing more optional parameters when integrated into CI systems.

Corporate WeChat Reminder

Based on the commandlet mechanism of ResScannerUE, it is very convenient to integrate corporate WeChat reminders. After submitting resources, the system automatically checks the submitted resources and sends group notifications:

The entire process only requires:

1. Python to launch the commandlet of ResScannerUE
2. Check the return value of the commandlet for any hit resources (returns -1)
3. Read the Saved/ResScanner/*_result.json file
4. Use Python to send corporate WeChat messages

Here, I provide an example:

resscanner_notice.py

import os
import sys
import chardet
import argparse
import codecs
import subprocess
from notice_robot_py import robot

commandlets_args = argparse.ArgumentParser(description="do commandlet")
commandlets_args.add_argument('--enginedir',help='engine root directory')
commandlets_args.add_argument('--projectdir',help='project root directory')
commandlets_args.add_argument('--projectname',help='project name,match projectname.uproject')
commandlets_args.add_argument('--config',help='config')
commandlets_args.add_argument('--notice',default="false",help='is enable notice to wechat')

wechat_robot_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
MentionList = [""]

def notice_to_wechat(Msg,bToWechat):
    URlStr = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=%s" % (wechat_robot_id)
    MsgType = "text"    
    if bToWechat:
        robot.Robot.set_robot_config(URlStr, MsgType, MentionList)
        res = robot.Robot.notice(Msg)
        if res is not None:
            print(res)

def get_args_by_name(parser_args,ArgName):
    args_pairs = parser_args.__dict__
    for key,value in args_pairs.items():
        if key == ArgName:
            return value

def ue_commandlet(cmd_params):
    parser_args = commandlets_args.parse_args()
    engine_dir = get_args_by_name(parser_args,"enginedir")
    project_dir = get_args_by_name(parser_args,"projectdir")
    project_name = get_args_by_name(parser_args,"projectname")

    engine_cmd_exe_path = "\"%s/Engine/Binaries/Win64/UE4Editor-cmd.exe\"" % (engine_dir)
    uproject_path = "\"%s/%s.uproject\"" % (project_dir,project_name)

    final_cmdlet = "%s %s %s" % (engine_cmd_exe_path,uproject_path,cmd_params)
    print(final_cmdlet)
    ps = subprocess.Popen(final_cmdlet)
    ps.wait()
    
    exit_code = ps.returncode # os.system(final_cmdlet)
    return exit_code

def main():
    parser_args = commandlets_args.parse_args()
    config = get_args_by_name(parser_args,"config")
    project_dir = get_args_by_name(parser_args,"projectdir")
    result_saved_dir = os.path.join(project_dir,"Saved","ResScanner","ResScanner_result.json")
    cmd_params = "-run=ResScanner -config=\"%s\"" % (config)
    
    exit_code = ue_commandlet(cmd_params)
    if exit_code != 0:
        if os.path.exists(result_saved_dir):
            with open(result_saved_dir, encoding='utf-8', errors='ignore') as f:
                s = f.read()
                lines = s.splitlines()
                for line in lines:
                    splited_line = line.rsplit(',',1)
                    if len(splited_line) == 2 and splited_line[1]:
                        match_role = splited_line[1]
                        match_role = match_role.strip()
                        if match_role not in MentionList:
                            MentionList.append(match_role)
                msg = "Triggered resource scan, having non-compliant resources\n%s" %(s)
                bIsNotice = str2bool(get_args_by_name(parser_args,"notice"))
                notice_to_wechat(msg,bIsNotice)
                f.close()

if __name__ == "__main__":
   main()

Usage method:

python3 resscanner_notice.py
--enginedir C:\Program Files\Epic Games\UE_4.26
--projectdir D:\UnrealProjects\Client 
--projectname PROJECT_NAME
--config D:\UnrealProjects\Client\ResScannerConfig.json
--notice true

By passing these parameters, you can launch ResScanner for scanning and notify the results to corporate WeChat.

Pre-Commit Hook

The previous automation process primarily triggered detection and report sending after submission. Essentially it’s a process of problem occurs first, problem solved later.

So, can we eliminate the contamination of erroneous resources to the trunk at an early stage, detecting whether resources are compliant before everyone submits? If non-compliant, submission should be prohibited.

The answer is yes. We can combine the GitChecker.bUncommitFiles provided by ResScannerUE (see Detection of Uncommitted Files) with Git’s Pre-Commit Hook mechanism to achieve:

1. Execute the Hook script during Commit
2. The Hook script launches the commandlet of ResScannerUE to conduct compliance scanning of uncommitted resources (GitChecker.bUncommitFiles = true)
3. Obtain the exit code of the commandlet. If there are non-compliant resources, stop the commit and prompt about the non-compliance.

The script to be executed by the Git Pre-Commit Hook is as follows:

res_rescanner_pre_commit_hook.py

#coding=utf-8
import os
import sys
import subprocess

# handle encoding convert
def conv_encoding(content_str):
    # return content_str.decode('utf-8').encode('gbk')
    return content_str

def main():
   engine_cmd_exe_path = "D:/UnrealEngine/Engine/Engine/Binaries/Win64/UE4Editor-cmd.exe"
   uproject_path = "D:/UnrealProjects/Client/FGame.uproject"
   config_path = "D:/UnrealProjects/Client/CommandletConfig/PRE_COMMIT_HOOK.json"
   cmd_params = "-run=ResScanner -config=\"%s\" -bDiffCommit=false -bUncommitFiles=true" % (config_path)
   result_saved_dir = "D:/UnrealProjects/Client/Saved/ResScanner/PRE_COMMIT_HOOK_result.json"

   final_cmdlet = "%s %s %s" % (engine_cmd_exe_path,uproject_path,cmd_params)
   FNULL = open(os.devnull, 'w')
   ps = subprocess.Popen(conv_encoding(final_cmdlet),stdout=FNULL,stderr=subprocess.STDOUT)
   ps.wait()
   exit_code = ps.returncode

   if exit_code != 0:
        if os.path.exists(result_saved_dir):
            with open(result_saved_dir) as f:
                s = f.read()
                lines = s.splitlines()
                for line in lines:
                    splited_line = line.rsplit(',',1)
                    if splited_line[1]:
                        MentionList.append(splited_line[1])
                msg = "This submission has non-compliant resources:\n%s" %(s.decode(encoding = "utf-8-sig").encode('utf-8'))
                print(conv_encoding(msg))
                f.close()
        else:
            print("%s file not found!" % (result_saved_dir))

   exit(exit_code)
if __name__ == "__main__":
   main()

Place it in the root directory of Content, create a pre-commit file under Content/.git/hooks, and fill it with the following sh script:

#!/bin/sh
python res_rescanner_pre_commit_hook.py
exitCode="$?"
exit $exitCode

Execution effect:

Combining with some Git GUI clients can achieve a friendly prompt effect:

Conclusion

This article introduces the engineering practice of automating resource scanning based on ResScannerUE, which can easily integrate resource scanning into various processes of the development pipeline.