Z's Blog

Use reflection to create properties cache for assets in UE

UE中利用反射为资产建立属性缓存
Posted on In , Views:
Symbols count in article: 8.6k Reading time ≈ 22 mins.

In the previous article Design and Implementation of Resource Self-Correction in UE, I introduced the implementation plan of filtering using the asset inspection of ResScannerUE, followed by automated processing.

Normally, if you want to check a certain property within a resource, you need to load the asset and retrieve the object:

However, if you want to check the properties of many resources in bulk, loading resources one by one in this way takes a long time, especially in the absence of DDC, where loading resources triggers the construction of DDC cache, which is time-consuming and has a high performance overhead.

As the size of the assets increases, scanning all resources in a project completely can be extraordinarily time-consuming. Therefore, I wonder if it’s possible to implement a method that does not require loading resources but can still retrieve properties within resources.

After research, I found a clever way to achieve this by combining reflection, asset serialization, and the features of AssetRegistry. Moreover, this is a non-intrusive approach that does not require modifying any existing resource type code.

This article will introduce the implementation principles and analyze the related logic in the engine, as well as present the practical application of this mechanism in the project.

Introduction

In my initial conception, one of my ideas was to cache the properties of resources during editing separately, recording the UUID value of the cached resource to verify the consistency between the resource and its cache, and then uniformly reading from this cache later.

However, I later abandoned this idea because changes to resources involve many people, and each person has a separate local runtime environment. How should the generated property caches be synchronized? If each machine generates independently, how should conflicts be resolved when multiple people edit the same resource? One problem leads to multiple potential issues, and in this case, it can only serve as a local temporary cache rather than a universal global data source.

Nevertheless, the overall idea of establishing a property cache for assets has not changed. The key is how to cache properties to be shared across the entire project. The process is as follows:

Moreover, it should ideally require no additional management or synchronization cost; the best approach would be completely non-intrusive.

Property Cache

The core part of this article lies in establishing the asset property cache and how to store it.

In UE, it’s possible to cache the values of certain properties. When you hover over a resource in the Content Browser, the asset’s basic information and some properties will pop up in AssetTips:
AnimSequence

Texture

The engine can obtain and display these properties and values without loading resources. These properties are marked as AssetRegistrySearchable in the C++ of the respective asset type. Taking UTexture as an example:

Engine/Texture.h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/** A bias to the index of the top mip level to use. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=LevelOfDetail, meta=(DisplayName="LOD Bias"), AssetRegistrySearchable)
int32 LODBias;

/** Compression settings to use when building the texture. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=Compression, AssetRegistrySearchable)
TEnumAsByte<enum TextureCompressionSettings> CompressionSettings;

/** The texture filtering mode to use when sampling this texture. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=Texture, AssetRegistrySearchable, AdvancedDisplay)
TEnumAsByte<enum TextureFilter> Filter;

/** The texture mip load options. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = Texture, AssetRegistrySearchable, AdvancedDisplay)
ETextureMipLoadOptions MipLoadOptions;

/** Texture group this texture belongs to */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=LevelOfDetail, meta=(DisplayName="Texture Group"), AssetRegistrySearchable)
TEnumAsByte<enum TextureGroup> LODGroup;

/** This should be unchecked if using alpha channels individually as masks. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=Texture, meta=(DisplayName="sRGB"), AssetRegistrySearchable)
uint8 SRGB:1;

They will be serialized into the uasset upon asset saving and will generate AssetData for the asset at engine startup, adding it to AssetRegistry for quick filtering of assets in the editor (for instance, to display only a certain type of resource), which is also the reason why assets can be quickly searched and filtered in the Content Browser.

I intend to utilize this feature to satisfy the demand for resource property caching.

Property Reflection FLAG

As mentioned earlier, the properties displayed in AssetTips are marked as AssetRegistrySearchable, so let’s analyze what the reflection marker UPROPERTY specifically does.

Taking CompressionSettings and SRGB from UTexture as examples:

1
2
3
4
5
6
/** Compression settings to use when building the texture. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=Compression, AssetRegistrySearchable)
TEnumAsByte<enum TextureCompressionSettings> CompressionSettings;
/** This should be unchecked if using alpha channels individually as masks. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category=Texture, meta=(DisplayName="sRGB"), AssetRegistrySearchable)
uint8 SRGB:1;

The reflection code generated from these reflection properties is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// CompressionSettings
const UE4CodeGen_Private::FBytePropertyParams Z_Construct_UClass_UTexture_Statics::NewProp_CompressionSettings = { 
    "CompressionSettings", 
    nullptr, 
    (EPropertyFlags)0x0010010000000005, 
    UE4CodeGen_Private::EPropertyGenFlags::Byte, 
    RF_Public|RF_Transient|RF_MarkAsNative, 
    1, 
    STRUCT_OFFSET(UTexture, CompressionSettings), 
    Z_Construct_UEnum_Engine_TextureCompressionSettings, 
    METADATA_PARAMS(Z_Construct_UClass_UTexture_Statics::NewProp_CompressionSettings_MetaData, UE_ARRAY_COUNT(Z_Construct_UClass_UTexture_Statics::NewProp_CompressionSettings_MetaData)) 
};

// SRGB
const UE4CodeGen_Private::FBoolPropertyParams Z_Construct_UClass_UTexture_Statics::NewProp_SRGB = { 
    "SRGB", 
    nullptr, 
    (EPropertyFlags)0x0010010000000005, 
    UE4CodeGen_Private::EPropertyGenFlags::Bool, 
    RF_Public|RF_Transient|RF_MarkAsNative, 
    1, 
    sizeof(uint8), 
    sizeof(UTexture), 
    &Z_Construct_UClass_UTexture_Statics::NewProp_SRGB_SetBit, 
    METADATA_PARAMS(Z_Construct_UClass_UTexture_Statics::NewProp_SRGB_MetaData, UE_ARRAY_COUNT(Z_Construct_UClass_UTexture_Statics::NewProp_SRGB_MetaData)) 
};

They share a common PropertyFlags value of 0x0010010000000005, which is a bitmask storing various FLAG values:

1
2
3
4
CPF_Edit                         = 0x0000000000000001,  ///< Property is user-settable in the editor.
CPF_BlueprintVisible             = 0x0000000000000004,   ///< This property can be read by blueprint code
CPF_AssetRegistrySearchable      = 0x0000010000000000,  ///< asset instances will add properties with this flag to the asset registry automatically
CPF_NativeAccessSpecifierPublic  = 0x0010000000000000,  ///< Public native access specifier

These are used to mark the purpose of the reflected property. In this example, it indicates: properties that are editable in the editor, displayed in blueprints, searchable by the AssetRegistry, and are public.

In the editor or other places requiring access to this reflection property, FLAGs can be used to determine whether a particular property should be processed.

Adding the AssetRegistrySearchable marker to the UPROPERTY allows the reflection property to acquire the CPF_AssetRegistrySearchable, enabling it to be identified by the AssetRegistry system and correctly serialized into the uasset.

Serialization

The previous section mentioned that by adding the AssetRegistrySearchable marker, it allows serialization into the uasset. This section will introduce how the engine implements the storage and reading.

Saving

When the engine saves a resource, it triggers the serialization of AssetRegistryData, with the call stack as follows:

During this process, it retrieves the TAG values of all Object properties marked as AssetRegistrySearchable in the current resource and serializes them into the asset:

The final serialized content of the Tag is:

It is an array in a Key-Value format, where the Key is the string name of the reflection property and the Value is the current resource value for that property, exported in the form of ExportTextureItem (similarly to the implementation ideas in Design and Implementation of Resource Self-Correction in UE#Universal Property Replacement Implementation), achieving universal storage of any property.

Reading

In the article Resource Management: UASSET Resource Encryption Scheme, I introduced content regarding resource encryption in UE, which includes a section on Package Summary.

In the Summary, there is a dedicated property used to mark the file offset of AssetRegistryData within the current asset:

Based on this offset, it is possible to directly access AssetRegistryData without fully loading a resource, simply by reading the PackageSummary.

In FPackageReader::ReadAssetRegistryData, it reads the uasset file and constructs FAssetData:

Then it adds this to AssetDataList:

1
2
3
4
5
6
7
8
9
10
11
12
// Create a new FAssetData for this asset and update it with the gathered data
AssetDataList.Add(
    new FAssetData(
        FName(*PackageName), 
        FName(*PackagePath), 
        FName(*AssetName), 
        FName(*ObjectClassName), 
        MoveTemp(TagsAndValues), 
        PackageFileSummary.ChunkIDs, 
        PackageFileSummary.PackageFlags
    )
);

This allows for direct retrieval of the cached resource’s AssetData from AssetRegistry.

Non-Intrusive Implementation

The previous sections introduced how to add the AssetRegistrySearchable reflection property, as well as the logic behind storing during resource saving and loading AssetData from uasset at engine startup.

In simple terms, by adding the AssetRegistrySearchable FLAG to properties, it allows for serialization into the asset without loading resources for access.

This achieves half of the requirements we previously outlined. However, there remains a crucial question: what to do with properties that do not have the AssetRegistrySearchable reflection marker added in the code?
If we could only manually add AssetRegistrySearchable to the required properties, we would have to modify its C++ code each time a new property is required. This is unacceptable; making invasive modifications is too limiting.

So, is there a method to dynamically add AssetRegistrySearchable to properties without modifying the code?

Yes! By utilizing UE’s reflection to dynamically modify the FProperty reflection FLAG in UClass, as I previously described in the article UE: Hook UObject#Modify UClass Control Reflection Property.

For the needs of this article, we need to add the CPF_AssetRegistrySearchable FLAG to FProperty:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
FProperty* GetPropertyByName(UClass* Class, FName PropertyName)
{
	FProperty* Property = nullptr;
	for(TFieldIterator<FProperty> PropertyIter(Class);PropertyIter;++PropertyIter)
	{
		FProperty* PropertyIns = *PropertyIter;
		if(PropertyIns->GetName().Equals(PropertyName.ToString()))
		{
			Property = PropertyIns;
		}
	}
	return Property;
}

void AddAssetRegistrySearchableToProperty(UClass* Class, const FString& PropertyName,
	bool bSearchable)
{
	FProperty* Property = GetPropertyByName(Class,*PropertyName);
	if(!Property->HasAnyPropertyFlags(EPropertyFlags::CPF_AssetRegistrySearchable))
	{
		if(bSearchable)
		{
			Property->SetPropertyFlags(EPropertyFlags::CPF_AssetRegistrySearchable);
		}
		else
		{
			Property->ClearPropertyFlags(EPropertyFlags::CPF_AssetRegistrySearchable);
		}
	}
}

Based on the analysis of serializing AssetRegistryData during resource saving, we simply need to modify the UClass of the resource type before actually saving it.

Taking BoneCompressionSettings from AnimSequence as an example, it does not have the AssetRegistrySearchable marker added:

1
2
3
/** The bone compression settings used to compress bones in this sequence. */  
UPROPERTY(Category = Compression, EditAnywhere, meta = (ForceShowEngineContent))  
class UAnimBoneCompressionSettings* BoneCompressionSettings;

During engine runtime, we can execute the following code:

1
AddAssetRegistrySearchableToProperty(UAnimSequence::StaticClass(),TEXT("BoneCompressionSettings"),true);

Then save the asset, in GetAssetRegistryTagFromProperty, it will detect that it has the CPF_AssetRegistrySearchable FLAG:

And it can export the actual property value as a string:

This allows us to perfectly meet our requirement to serialize certain properties into the asset.

Reuse and Consistency

Since AssetData is directly serialized into the asset, it will be updated with each resource saving. Therefore, as long as the latest resource can be updated, the accuracy of its property values can be ensured.

Moreover, for collaboration, each person will save their modified resources. If conflicts arise between resources from different submissions, version control can resolve this. This can ensure consistency between the resources in the remote repository and their cached property values (all stored within a single uasset file), preventing confusion.

Caching Property Reading

Now that the property caching for assets has been fully implemented, how can it be applied?

In the application of ResScanner:

1
2
3
4
5
6
7
8
9
auto GetPropertyValueByName = [&Rule,this](const FAssetData& AssetData,const FString& PropertyName)  
{  
    FString AssetDataTagValue;  
    if(!AssetData.GetTagValue(*PropertyName,AssetDataTagValue))  
    {       
        AssetDataTagValue = UFlibAssetParseHelper::GetPropertyValueByName(GetPropertyOwning(AssetData,Rule),PropertyName);  
    }    
    return AssetDataTagValue;  
};

The resources are only loaded when property values cannot be obtained from TagsAndValues.

For ResScannerUE, there may be many rules for property inspection. Before the scanning module starts, all rules containing property checks will be automatically retrieved, and CPF_AssetRegistrySearchable will be added to the properties to be checked. Regardless of how many rules are added, it can achieve a non-intrusive effect.

Conclusion

Some of the knowledge involved in this article is summarized and analyzed more specifically in other articles on the blog and can serve as extended reading.

This article analyzes and utilizes engine features such as type reflection, asset serialization, and AssetRegistry data generation, and combines them to implement a non-intrusive mechanism for caching resource properties without modifying any existing resource type code.

By allowing access to properties within resources without loading the resources, the demand for “resource property checks” will not lead to significant increases in check duration due to the expansion of resource scale, nor will it generate issues during scans when the local environment lacks DDC, thus greatly enhancing the checking efficiency.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:Use reflection to create properties cache for assets in UE
Author:LIPENGZHA
Publish Date:2023/10/25 15:27
World Count:8.6k Words
Link:https://en.imzlp.com/posts/71162/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!