SARC: Difference between revisions

From ZeldaMods (Breath of the Wild)
Jump to navigation Jump to search
imported>Leoetlino
 
(10 intermediate revisions by 2 users not shown)
Line 3: Line 3:
== Structure ==
== Structure ==
The basic structure of SARC files is documented on the [[mk8:SARC (File Format)|MK8 wiki]].
The basic structure of SARC files is documented on the [[mk8:SARC (File Format)|MK8 wiki]].
=== Hash algorithm ===
Nintendo's algorithm uses an unsigned 32-bit integer for the hash variable. Also note that Nintendo iterates over each byte, not each string character.
Here is an accurate implementation of the algorithm, written in C++:
<source lang="c++">
// key is equal to 0x65
u32 calcHash32(const sead::SafeString& str, u32 key)
{
    const char* str_ = str.cstr();
    u32 result = 0;
    // Each character must be treated as a signed value.
    // The cast to s8 (not s32) is necessary to avoid unsigned conversions.
    for (s32 i = 0; str_[i] != '\0'; i++)
        result = result * key + s8(str_[i]);
    return result;
}
</source>
Note that in older versions of sead, characters were not explicitly casted to <code>s8</code> so the end result would depend on the signedness of <code>char</code> (which is implementation-defined). For GHS compiled code on the Wii U, <code>char</code> is unsigned whereas it's signed with Clang (AArch64 target). PC tools most likely use a signed <code>char</code> as that is the default signedness for x86 targets.
In newer versions of sead, an explicit cast to <code>signed char</code>, aka <code>std::int8_t</code>, aka <code>s8</code>, was added to ensure hashes are the same regardless of the target platform. Without the cast, non-ASCII characters (e.g. Japanese UTF-8 bytes) would result in hashes being different across platforms.


== Usage in ''Breath of the Wild'' ==
== Usage in ''Breath of the Wild'' ==
Line 34: Line 59:
Because of the alignment problem and file size limitations due to ''Breath of the Wild'''s [[Resource system|resource system]], only the following tools are recommended:
Because of the alignment problem and file size limitations due to ''Breath of the Wild'''s [[Resource system|resource system]], only the following tools are recommended:


* [https://github.com/aboood40091/SARC-Tool SARC Tool] (for Wii U and Switch) '''Warning''': SARC Tool may add unnecessary padding when archives are repacked, meaning generated archives may end up larger than the original even if no file size was increased inside the SARC. ''The game will fail to load it unless you fix the [[RSTB]]''.
{{tool table|category=Tools (SARC)|notcategory=Unrecommended tools}}
* [https://github.com/Shadsterwolf/BotWUnpacker BotW Unpacker] (for Wii U)
* [https://github.com/leoetlino/sarc <code>sarc</code>] (for Wii U and Switch)


'''We strongly advise against using other tools because they do not handle file alignment properly.'''
'''We strongly advise against using other tools because they do not handle file alignment properly.'''


[[Category:File formats]] 
[[Category:File formats]] 

Latest revision as of 23:45, 13 January 2021

SARCs are archive files.

Structure

The basic structure of SARC files is documented on the MK8 wiki.

Hash algorithm

Nintendo's algorithm uses an unsigned 32-bit integer for the hash variable. Also note that Nintendo iterates over each byte, not each string character.

Here is an accurate implementation of the algorithm, written in C++:

// key is equal to 0x65
u32 calcHash32(const sead::SafeString& str, u32 key)
{
    const char* str_ = str.cstr();

    u32 result = 0;
    // Each character must be treated as a signed value.
    // The cast to s8 (not s32) is necessary to avoid unsigned conversions.
    for (s32 i = 0; str_[i] != '\0'; i++)
        result = result * key + s8(str_[i]);

    return result;
}

Note that in older versions of sead, characters were not explicitly casted to s8 so the end result would depend on the signedness of char (which is implementation-defined). For GHS compiled code on the Wii U, char is unsigned whereas it's signed with Clang (AArch64 target). PC tools most likely use a signed char as that is the default signedness for x86 targets.

In newer versions of sead, an explicit cast to signed char, aka std::int8_t, aka s8, was added to ensure hashes are the same regardless of the target platform. Without the cast, non-ASCII characters (e.g. Japanese UTF-8 bytes) would result in hashes being different across platforms.

Usage in Breath of the Wild

SARCs are used extensively in Breath of the Wild to keep related data in memory and minimise loading times.

Extensions

Archives have a wide range of extensions. However, the file format is completely the same regardless of the extension.

The following extensions are specific to the game:

The following extensions are used by Nintendo libraries that are included in the game (non exhaustive list):

  • sarc
  • bgenv
  • genvb
  • blarc

Data alignment

Some files have specific alignment requirements (e.g. for GPU data). Because Nintendo's SARC library returns file data by giving pointers to the data section directly, special care must be taken to pack files in a way that satisfies all alignment requirements.

Nintendo libraries do not use BotW's resource system and expect files to be properly aligned. This is the case for layout archives (blarc) and agl environment files (Pack/Bootup.pack/Env/env.genvb).

However unlike most other Nintendo games, for files that are managed by the game's resource system, aligning archive file data is usually unnecessary because the system will automatically allocate an aligned buffer and copy the archive data into it.

Tools

Because of the alignment problem and file size limitations due to Breath of the Wild's resource system, only the following tools are recommended:

Tool Cross-platform Setup Known issues
SARC Tool Yes
  1. Install the sarclib package. [CLI help]
  2. Install the libyaz0 package. [CLI help]
  3. Download the latest release from GitHub
  • Unnecessary padding for BotW: SARC Tool may add unnecessary padding when archives are repacked, meaning generated archives may end up larger than the original even if no file size has increased inside the SARC. The game will fail to load it unless you fix the Resource system.
sarc Yes

Install the sarc package. [CLI help]

None

BotW Unpacker Yes

Download the latest release from GitHub.

  • Wii U only: This tool can only handle big endian archives. Switch SARCs will not work as they use little endian.
Wild Bits Yes
  1. Ensure you have a 64 bit version of Python 3.7+ before continuing. Windows users must also install the latest x64 Visual C++ redistributable. You will not be able to install or launch Wild Bits otherwise. See Help:Setting_up_tools.
  2. (Optional) For best results on Windows, install cefpython3. [CLI help]
  3. Install the wildbits package. [CLI help]

On some system configurations, launching with wildbits doesn't work and you must use python -m wildbits instead.

Switch Toolbox No
  • Simply download the latest compiled release from GitHub, unzip, and launch the exe. Note that the latest release has an old date, but it is current, as it automatically rebuilds with each commit.

We strongly advise against using other tools because they do not handle file alignment properly.