Z's Blog

ASTC texture compression analysis and efficiency optimization in UE

UE中ASTC贴图压缩分析及效率优化
Posted on In , Views:
Symbols count in article: 8.4k Reading time ≈ 21 mins.

ASTC stands for Adaptive Scalable Texture Compression, which is a popular texture compression scheme on mobile platforms. When the platform uses ASTC packaging, UE defaults to using the Intel ISPC Texture Compressor for texture compression. However, it has some limitations, only supporting compression specifications of 8x8 and above, while 10x10 and 12x12 are not supported. If specified in the project, it will still use the 8x8 specification. In addition to ISPC, the engine also provides ARM’s astc-encoder compression method, which supports specifications below 8x8, but is not enabled by default. Moreover, the integrated compression efficiency in the engine is very low, and using astc-encoder to compress textures can pose a significant challenge to Cook time in large-scale resources.

This article analyzes the settings for using ASTC compression for textures in UE, as well as the compression efficiency and optimization ideas for the astc-encoder in the engine.

Texture Compression Configuration

In Project Settings-Cooker-Texture, you can set the default resource quality and size levels: CookerSettings.h

Comparison with ASTC block sizes:

1
2
3
4
5
0=12x12 
1=10x10 
2=8x8 
3=6x6 
4=4x4

The default value in the engine is 3: Engine/Config/BaseEngine.ini

In the Texture resource editing, you can also set a specific Texture individually:

Lowest->Highest corresponds to values 0-4, using Default will follow the settings in the project configuration.

Also, the type of Compression Settings you set will affect the type of resource compression; Default refers to the parameters in project settings, while setting it to NormalMap type will result in ASTC_4x4.

Intel ISPC

Intel ISPC is the default ASTC compressor in the engine, but it only supports a maximum block size of 8x8; higher sizes like 10x10 and 12x12 are not supported.

Its GitHub README introduces:

ASTC (LDR, block sizes up to 8x8)

There is also a specific description in the Adaptive Scalable Texture Compression User Guide:

The Intel ISPC Texture Compressor is a compression library developed by Intel, supporting multiple texture formats including ASTC. It uses the Intel ISPC (Implicit SPMD Program Compiler) compiler to parallelize key parts of the compressor implementation for the target CPU SIMD instruction set.
The ASTC support in the tool only supports the 2D LDR profile and block sizes up to 8x8. It has image quality that is similar to astcenc -fast for photographic imagery, but can be up to three times faster at compressing. Its quality is measurably worse than astcenc -fast for non-photographic imagery, such as normal maps, mask maps, or cartoon-like color data.

This means that in UE, using ASTC with project configuration settings set above 8x8 will be ineffective, because the ISPC compression method only supports a maximum of 8x8.

The implementation in the UE code is the same: Engine/Source/Developer/TextureFormatIntelISPCTexComp/Private/TextureFormatIntelISPCTexComp.cpp

ARM astcenc

UE also provides another set of ASTC Texture compression: ARM’s astc-encoder. If ISPC is not used, it will utilize this.

1
2
3
4
5
6
// Developer\TextureFormatASTC\Private\TextureFormatASTC.cpp
#if PLATFORM_WINDOWS || PLATFORM_LINUX || PLATFORM_MAC
	#define SUPPORTS_ISPC_ASTC	1
#else
	#define SUPPORTS_ISPC_ASTC	0
#endif

Engine code: TextureFormatASTC.cpp#L450

By default, in UE4 versions, you can only switch this off by modifying the value of the SUPPORTS_ISPC_ASTC macro definition or modifying the engine’s CompressImage code implementation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static bool UseArmASTCCompressor()
{
	bool bUseArmASTC = false;
	GConfig->GetBool(TEXT("/Script/UnrealEd.CookerSettings"), TEXT("UsArmASTCCompressor"), bUseArmASTC, GEngineIni);
	return bUseArmASTC;
}
// for CompressImage
#if SUPPORTS_ISPC_ASTC
if (!UseArmASTCCompressor())
{
	return IntelISPCTexCompFormat.CompressImage(InImage, BuildSettings, bImageHasAlphaChannel, OutCompressedImage);
}
#endif
// arm astcenc...

Editing DefaultEngine.ini allows for control of the switch:

1
2
[/Script/UnrealEd.CookerSettings]
UsArmASTCCompressor=true

When ISPC compression is disabled, the alternative ARM astcenc will invoke the engine’s Binaries/ThirdParty directory’s astsenc program to compress textures:

Call stack:

The separate process execution is not executed within the UE process. During the execution of astcenc for compression, the default Cook process in UE operates on a per-resource basis; while the DDC cache process can be asynchronous, from the Cook perspective, the current resource remains blocking execution. Moreover, because it invokes a separate process for handling, there may be significant overhead from process startups and switches, making this process very slow. It may take several minutes even for compressing a single texture, posing a substantial challenge to build time.

The execution of astcenc constructs the command line parameters based on the texture settings:

1
2
3
4
# 4x4
../../../Engine/Binaries/ThirdParty/ARM/Win32/astcenc.exe -c "D:/UnrealProjects/AstcExample/Intermediate/Cache/60c572f7-471422f1-24309fb0-45c3d4f9-RGBToASTCIn.webp" "D:/UnrealProjects/AstcExample/Intermediate/Cache/60c572f7-471422f1-24309fb0-45c3d4f9-RGBToASTCOut.astc" 4x4 -thorough  -esw bgr1 -ch 1 1 1 0 "
# 12x12
../../../Engine/Binaries/ThirdParty/ARM/Win32/astcenc.exe -c "D:/UnrealProjects/AstcExample/Intermediate/Cache/ea8ed05a-4371fffd-333d47a1-cf7b6490-RGBToASTCIn.webp" "D:/UnrealProjects/AstcExample/Intermediate/Cache/ea8ed05a-4371fffd-333d47a1-cf7b6490-RGBToASTCOut.astc" 12x12 -thorough -normal_percep -esw bbbg -dsw rgba"

Invoking multiple processes compresses different Mip levels of the textures separately:

Each Mip corresponds to a separate astcenc processing process. Call stack:

Note: The compression parameters for different Mip levels are exactly the same, but the images being processed for compression are different. This is something to note, which can also be seen from the call stack above.

UE5

In UE5, ASTC default is also the ISPC compressor, but can be controlled through cook.ASTCTextureCompressor, where 0 is ISPC and 1 is ARM.

Additionally, the default compressor in Project Settings-Cooker can be configured:

Efficiency Analysis & Optimization

DDC Log

Analyzing the texture compression process, you can enable Log and LogTexture Verbose logging:

1
2
3
4
; DefaultEngine.ini
[Core.Log]
LogDerivedDataCache=Verbose
LogTexture=Verbose

During the Cook process, the following information will be output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
LogDerivedDataCache: Verbose: GetSynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) Cache miss on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogTexture: Display: Building textures: T_ForestGround_D (ASTC_RGBAuto, 2048X2048)
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532 for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716 for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogTexture: Verbose: Storing texture in DDC:
  Key: TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B
  Format: ASTC_6x6
  Mip0 2048x2048 1871424 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048
  Mip1 1024x1024 467856 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024
  Mip2 512x512 118336 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512
  Mip3 256x256 29584 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256
  Mip4 128x128 7744 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128
  Mip5 64x64 1936 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP5_64x64
  Mip6 32x32 576 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP6_32x32
  Mip7 16x16 144 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP7_16x16
  Mip8 8x8 64 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP8_8x8
  Mip9 4x4 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP9_4x4
  Mip10 2x2 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP10_2x2
  Mip11 1x1 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP11_1x1
  Derived Data: 4648 bytes
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19506
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19507
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19508
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19509
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19510
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19506
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19506, SUCCESS
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19507
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19507, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19508
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19508, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19510
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19510, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19510, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716 to ../../../Engine/DerivedDataCache/7/7/2/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125.udd
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19508, SUCCESS
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19509
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19509, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19510
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A to ../../../Engine/DerivedDataCache/5/8/2/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532 to ../../../Engine/DerivedDataCache/2/1/5/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312 to ../../../Engine/DerivedDataCache/2/6/1/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532

It facilitates viewing detailed information during the DDC and texture compression process.

Main Time Consumption

Based on the previous analysis, while using ARM astcenc for texture compression during Cook, the overhead for a single texture roughly consists of the following parts:

  1. Serialization of the file (noting that it needs to be saved/read during compression)
  2. Cold start of the compression process
  3. Waiting for compression to complete (compression process exit)
  4. Multiple Mip levels can start multiple processes

In the original Cook process of UE, which handles resources one at a time, it cannot effectively utilize the multi-core advantages of hardware. Essentially, it cooks one texture at a time, compressing it Mip by Mip, resulting in a serial loop process.

Using HotPatcher for a single texture ARM astcenc test takes anywhere from a few seconds to several minutes:

1
2
3
Display: Package /Game/ExampleContent/Landscapes/Textures/T_ForestGround_D Time : 29.767000s
Display: Package /Game/ExampleContent/Landscapes/Textures/T_ForestGround_N Time : 145.753000s
Display: Package /Game/ExampleContent/ScriptExamples/PivotPainter/Textures/T_PalmTree_N Time : 232.641000s

Cooking 28 textures took 14 minutes:

Textures are the largest resource type in the project; in the case of a complete DDC miss, the Cook time is unimaginable, reaching several dozen hours.### Replacing astcenc

In UE4, the engine’s default precompiled astcenc is a very old version 1.3:

1
2
3
ASTC codec version 1.3
Copyright (C) 2011-2013 ARM Limited
All rights reserved. Use of this software is subject to terms of its license.

Moreover, it is 32-bit and lacks SSE/SSE2 instruction set optimizations:

1
2
3
4
5
6
7
8
9
10
FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        515B0518 time date stamp Wed Apr  3 00:19:36 2013
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

Upgrading astcenc Version

Looking at the release notes on Github and some articles, newer versions of astcenc have shown significant improvements in compression performance.

In UE5, the integrated astcenc is 3.2.0, and it is a 64-bit version with SSE instruction set optimizations. However, the astcenc in UE5 cannot be directly used in UE4; there are significant differences in parameters between versions, necessitating the porting of UE5’s implementation to UE4, aligning the old engine’s constructed astcenc parameters with the new version.

Compile x64 SSE Version

This method allows for maintaining compatibility of the engine’s astcenc parameters without altering the engine’s implementation, simply requiring the compilation of a 64-bit SSE instruction set optimized version.

Note: When compiling 64-bit, SSE instruction set optimization is enabled by default, and there’s no need to specify /arch:sse separately.

Before Optimization:

After Optimization:

From the original 7m12s to 1m24s, saving about 80% of the time—a substantial improvement.

Win32 Win64 SSE
Time (s) 432 84

Parallel Cache DDC

Replacing astcenc can improve the compression efficiency of a single texture, but the process of handling resources one at a time still limits performance. Even if all Mips are processed in parallel, the maximum number of astcenc processes will correspond to the number of Mips, and it cannot fully utilize the performance of high-core CPUs.

Therefore, the Cooking process for a single resource can be broken down, caching resources’ DDC in batches without performing serialization operations. By using asynchronous tasks to launch astcenc for the actual compression process, an individual UE process can simultaneously handle multiple texture compression tasks.

This can fully leverage the performance of multi-core CPUs while avoiding the inefficiencies of executing per-resource and per-Mip processes, where performance is not maximized and the CPU idles.

However, under this single Cook process scenario, parallel compressing textures requires a higher performance from the machine’s CPU. In simple terms, if a 4-core CPU is tasked with running significantly more tasks than its core count, it will not only fail to accelerate the process but may slow down due to frequent context switching (increased system process scheduling overhead, competition for CPU among processes, etc.).

A reasonable approach is to determine the number of tasks based on the core count to utilize the CPU fully while avoiding overload. However, the physical performance of a machine is ultimately limited, making it difficult to balance massive resources and local machine bottlenecks.

Thus, I implemented a multi-machine parallel Cook solution, differing from implementations like FastBuild and IB that are limited to code and Shader compilation and cannot be applied to other resource types. In this case, the time for texture compression has already far exceeded the time for Shader compilation.

Single machine operation simulation:

In multi-machine mode, each Agent can allocate a certain amount of resources based on machine performance, avoiding the limitations of a single machine’s capabilities. Moreover, machines can share the DDC data produced by each other, achieving true parallel Cook.

The article is finished. If you have any questions, please comment and communicate.

Scan the QR code on WeChat and follow me.

Title:ASTC texture compression analysis and efficiency optimization in UE
Author:LIPENGZHA
Publish Date:2022/09/08 19:01
World Count:8.4k Words
Link:https://en.imzlp.com/posts/7181/
License: CC BY-NC-SA 4.0
Reprinting of the full article is prohibited.
Your donation will encourage me to keep creating!