Resource system - Revision history

imported>Leoetlino at 14:49, 20 March 2020

2020-03-20T14:49:40Z

← Older revision		Revision as of 14:49, 20 March 2020
Line 1:		Line 1:
	In ''Breath of the Wild'', the '''resource system''' is a subsystem that is responsible for managing resources, i.e. content files. It is composed of [[ResourceMgrTask]], [[ResourceSystem]] and a few ancillary classes (such as "resource binders" and "handles").		In ''Breath of the Wild'', the '''resource system''' is a subsystem that is responsible for managing resources, i.e. content files. It is composed of [[ResourceMgrTask]], [[ResourceSystem]] and a few ancillary classes (such as "resource binders" and "handles").

			BotW's resource system is also used in ''[[wikipedia:Animal Crossing: New Horizons\|Animal Crossing: New Horizons]]'' and called "ares" or "ARes".

	== Concepts ==		== Concepts ==

imported>Leoetlino: /* Resource size calculation */ mention how the table is automatically generated by Nintendo

2020-02-10T08:29:23Z

Resource size calculation: mention how the table is automatically generated by Nintendo

← Older revision		Revision as of 08:29, 10 February 2020
Line 110:		Line 110:

	==== Resource size calculation ====		==== Resource size calculation ====
	~~It is currently unknown how~~ Nintendo ~~has determined~~ the ~~values~~ in the RSTB~~. However by REing~~ the ~~resource system it was found~~ that ~~the~~ resource loading heap is, for most factories, only used to allocate the file loading buffer (which is as large as the file to load), the C++ resource class, extra allocations depending on the resource class and some extra bytes to ensure data is aligned correctly in memory.		According to an internal source, Nintendo automatically generates the Resource Size Table using development builds of the game that are believed to contain instrumentation code (perhaps in the [[Patrol]] component) This is consistent with the fact that the RSTB asset has the "product" suffix in its name, which is used for machine-generated files, and also with the fact that a similar solution is used for actor instance heap size measurement.

			The resource loading heap is, for most factories, only used to allocate the file loading buffer (which is as large as the file to load), the C++ resource class, extra allocations depending on the resource class and some extra bytes to ensure data is aligned correctly in memory.

	Therefore, the following formula should give a correct size value for all resource types:		Therefore, the following formula should give a correct size value for all resource types:
Line 117:		Line 119:

	PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.		PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.


	[[Category:Internals]]		[[Category:Internals]]
	<references />		<references />

imported>NiceneNerd: Moved BFRES RSTB estimation to Help:Updating the RSTB

2019-08-01T05:05:39Z

Moved BFRES RSTB estimation to Help:Updating the RSTB

@@ Line 117: / Line 117: @@
 PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.
 [[Category:Internals]]
 <references />

imported>Ginger: Clarified BFRES resource size estimation for Wii U

2019-08-01T04:42:16Z

Clarified BFRES resource size estimation for Wii U

← Older revision		Revision as of 04:42, 1 August 2019
Line 118:		Line 118:
	PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.		PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.

	==== BFRES resource size estimation ====		==== BFRES resource size estimation (Wii U only) ====
	[[File:Allrstb.png\|alt=All BFRES RSTB entries\|thumb\|RSTB size analysis for all BFRES files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Allrstb.png\|alt=All BFRES RSTB entries\|thumb\|RSTB size analysis for all BFRES files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
	Though current tools cannot yet calculate exact RSTB values for BFRES files, by analyzing the BFRES files Nintendo shipped with the game and their RSTB entries, it is possible to make some general observations which permit loose estimates for setting BFRES RSTB values.		Though current tools cannot yet calculate exact RSTB values for BFRES files, by analyzing the BFRES files Nintendo shipped with the game and their RSTB entries, it is possible to make some general observations which permit loose estimates for setting BFRES RSTB values.
Line 140:		Line 140:

	[[File:Tex2rstb.png\|thumb\|RSTB size analysis for BFRES tex2 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Tex2rstb.png\|thumb\|RSTB size analysis for BFRES tex2 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
			Note: These analyses were done on the Wii U 1.5.0 version. Switch version calculations will vary.

imported>NiceneNerd: Grammar and style

2019-08-01T04:36:38Z

Grammar and style

← Older revision		Revision as of 04:36, 1 August 2019
Line 120:		Line 120:
	==== BFRES resource size estimation ====		==== BFRES resource size estimation ====
	[[File:Allrstb.png\|alt=All BFRES RSTB entries\|thumb\|RSTB size analysis for all BFRES files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Allrstb.png\|alt=All BFRES RSTB entries\|thumb\|RSTB size analysis for all BFRES files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
	Though exact RSTB ~~entry sizes are currently unable to be calculated exactly~~, by analyzing the BFRES files Nintendo shipped with the game, ~~a few revelations have been made~~ to ~~allow~~ for ~~acceptable~~ RSTB ~~entry size estimations~~.		Though current tools cannot yet calculate exact RSTB values for BFRES files, by analyzing the BFRES files Nintendo shipped with the game and their RSTB entries, it is possible to make some general observations which permit loose estimates for setting BFRES RSTB values.

	There are two types of BFRES file: model, and texture. (Texture is further ~~split~~ into ~~main~~ textures, ~~tex1~~, and mipmaps, ~~tex2~~, but there seems to be no difference in RSTB calculations concerning these two types of textures)		There are two types of BFRES file: model and texture. (Texture files are further divided into regular textures, which include <code>Tex1</code> in the name, and mipmaps, which contain <code>Tex2</code> in the name, but there seems to be no difference in RSTB calculations concerning these two types of textures.)

	[[File:Modelrstb.png\|thumb\|RSTB size analysis for BFRES model files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Modelrstb.png\|thumb\|RSTB size analysis for BFRES model files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
Line 133:		Line 133:

	[[File:Tex1rstb.png\|thumb\|RSTB size analysis for BFRES tex1 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Tex1rstb.png\|thumb\|RSTB size analysis for BFRES tex1 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
	For textures, the ratio of RSTB entry size to BFRES file size increases late, but sharply, as BFRES file size decreases. This makes the estimates for RSTB entry size for textures incredibly easy		For textures, the ratio of RSTB entry size to BFRES file size increases late, but sharply, as BFRES file size decreases. This makes the estimates for RSTB entry size for textures incredibly easy.

	* If your texture BFRES is less than 100KB, your RSTB entry should be 200% of your BFRES file size.		* If your texture BFRES is less than 100KB, your RSTB entry should be 200% of your BFRES file size.

imported>Ginger: Cleaned up whitespace in BFRES resource size estimation section.

2019-08-01T04:30:03Z

Cleaned up whitespace in BFRES resource size estimation section.

← Older revision		Revision as of 04:30, 1 August 2019
Line 124:		Line 124:
	There are two types of BFRES file: model, and texture. (Texture is further split into main textures, tex1, and mipmaps, tex2, but there seems to be no difference in RSTB calculations concerning these two types of textures)		There are two types of BFRES file: model, and texture. (Texture is further split into main textures, tex1, and mipmaps, tex2, but there seems to be no difference in RSTB calculations concerning these two types of textures)





	~~<br />~~
	[[File:Modelrstb.png\|thumb\|RSTB size analysis for BFRES model files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Modelrstb.png\|thumb\|RSTB size analysis for BFRES model files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
	For models, the ratio of RSTB entry size to BFRES file size increases early, but lazily, as BFRES file size decreases. The estimations are trickier for models than they are for textures, but they are still fairly simple overall.		For models, the ratio of RSTB entry size to BFRES file size increases early, but lazily, as BFRES file size decreases. The estimations are trickier for models than they are for textures, but they are still fairly simple overall.
Line 137:		Line 132:
	* If your model BFRES is greater than 2MB, your RSTB entry should be 125% of your BFRES file size.		* If your model BFRES is greater than 2MB, your RSTB entry should be 125% of your BFRES file size.





	~~<br />~~
	[[File:Tex1rstb.png\|thumb\|RSTB size analysis for BFRES tex1 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]		[[File:Tex1rstb.png\|thumb\|RSTB size analysis for BFRES tex1 texture files. Y-axis: ratio, X-axis: BFRES file size, dot size: RSTB entry size.]]
	For textures, the ratio of RSTB entry size to BFRES file size increases late, but sharply, as BFRES file size decreases. This makes the estimates for RSTB entry size for textures incredibly easy		For textures, the ratio of RSTB entry size to BFRES file size increases late, but sharply, as BFRES file size decreases. This makes the estimates for RSTB entry size for textures incredibly easy
Line 159:		Line 149:


	~~<br />~~
	[[Category:Internals]]		[[Category:Internals]]
	<references />		<references />

imported>Ginger: Added BFRES resource size estimation and analysis.

2019-08-01T04:28:50Z

Added BFRES resource size estimation and analysis.

@@ Line 21: / Line 21: @@
 Note that:
 * For ResourceLoadArg3 (used in model/bfres related code), the factory is hardcoded to be the ResourceBase factory.
 * For ResourceLoadArg2 (used for actor resources and physics stuff) and ResourceLoadArg (everything else), the factory is determined from the file extension.
@@ Line 54: / Line 55: @@
 ==== Cases where the AoC file device is used ====
 If the AoC version is >= 0x200 (in most cases) or >= 0x300 (for DLC Pack 2 content), and if the canonical resource name (without the <code>Aoc/0010/</code> prefix) matches any of the below patterns:
 * <code>Terrain/A/AocField*</code>
 * Any dungeon pack (<code>Pack/%s.pack</code>): shrines, divine beasts, Final Trial (for shrines, the dungeon number must be > 119)
@@ Line 72: / Line 74: @@
 * <code>Voice/*/Stream_Demo6*/*.bfstm</code>
 * <code>System/AocVersion.txt</code>
 There are two other situations where the <code>Aoc/0010/</code> prefix is supposed to be prepended:
 * If the load file device is set to the <code>Pack/AocMainField.pack</code> archive file device explicitly
 * If the global resource pack is an AoC dungeon pack
 However, the first case is irrelevant because that archive only contains Map/MainField/ (a case that is already handled by the path checks) and the second check appears to never pass.
@@ Line 84: / Line 89: @@
 === Heap size ===
 The size of the resource loading heap the system allocates every time a resource is loaded depends on the value that is listed in the [[#Resource size table]] (RSTB). If lookup fails, the game will fall back to the following formula (Switch on 1.5.0):
 <syntaxhighlight lang="C++">
 alignedFileSize = (actualFileSize + 31) & -32;
@@ Line 95: / Line 99: @@
 + 0x750
 </syntaxhighlight>
 '''Warning''': Failure to add entries to the RSTB for a large number of resources may result in system instability. The purpose of this fallback appears to be to allow loading resources during development even without an entry in the resource size table. As such, the system will often allocate way more memory than needed to account for any dynamic allocation that the resource class may do.
@@ Line 110: / Line 113: @@
 Therefore, the following formula should give a correct size value for all resource types:
   (size rounded up to multiple of 32) + CONSTANT + sizeof(ResourceClass) + PARSE_SIZE
 CONSTANT is 0x168 in the Switch version and 0xe4 on Wii U.
 PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.
 [[Category:Internals]]

imported>Leoetlino at 17:43, 29 December 2018

2018-12-29T17:43:38Z

← Older revision		Revision as of 17:43, 29 December 2018
Line 118:		Line 118:

	[[Category:Internals]]		[[Category:Internals]]
	~~[[Category:Subsystems (BotW)]]~~

imported>Leoetlino: /* Resource size table */

2018-10-14T11:55:13Z

Resource size table

← Older revision		Revision as of 11:55, 14 October 2018
Line 116:		Line 116:

	PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.		PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.

	~~==== Editing tools ====~~
	~~{{tool table\|category=Tools (RSTB)}}~~

	[[Category:Internals]]		[[Category:Internals]]
	[[Category:Subsystems (BotW)]]		[[Category:Subsystems (BotW)]]

imported>Leoetlino: /* Resource size table */

2018-10-14T11:54:56Z

Resource size table

← Older revision		Revision as of 11:54, 14 October 2018
Line 116:		Line 116:

	PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.		PARSE_SIZE is the amount of memory allocated from the resource heap in the <code>Resource::parse</code> function. Determining the exact value of PARSE_SIZE requires reversing that function and tracking calls to <code>operator new()</code> because the amount of dynamically allocated memory depends on the resource.

			==== Editing tools ====
			{{tool table\|category=Tools (RSTB)}}

	[[Category:Internals]]		[[Category:Internals]]
	[[Category:Subsystems (BotW)]]		[[Category:Subsystems (BotW)]]