3dbrew

Extdata: Difference between revisions

← Older edit
Wwylele (talk | contribs)
242 edits
→Tools: old extdata_tool is a dead link; Add new tools
TimmSkiller (talk | contribs)
Trusted
843 edits
 
(18 intermediate revisions by 8 users not shown)
Line 29: Line 29:
* Device file with <code>SubDirID = SubFileID = 00000000</code> doesn't exist. Other ID combinations can exists.
* Device file with <code>SubDirID = SubFileID = 00000000</code> doesn't exist. Other ID combinations can exists.
* Device file with <code>SubDirID = 00000000</code> and <code>SubFileID = 00000001</code> is the VSXE metadata file and must exist.
* Device file with <code>SubDirID = 00000000</code> and <code>SubFileID = 00000001</code> is the VSXE metadata file and must exist.
* Other files, besides <code>Quota.dat</code> and <code>00000000/00000001</code>, are normal sub files, are these device files one-to-one correspond to virtual files.
* Other files, besides <code>Quota.dat</code> and <code>00000000/00000001</code>, are normal sub files, are these device files one-to-one correspond to virtual files. They contain raw virtual file data in the DIFF inner content.
* <code>SubDirID = 00000000</code> is usually the only one device directory that can be seen. See [[#Device Directory Capacity]] for more information.
* <code>SubDirID = 00000000</code> is usually the only one device directory that can be seen. See [[#Device Directory Capacity]] for more information.


== Quota File ==
== Quota File ==


The inner data of <code>Quota.dat</code> is 0x48 bytes with the following format. The exact function of this file is unclear.
The inner data of <code>Quota.dat</code> is 0x48 bytes with the following format. The file applies limits to the file system, similar to quotas for other file systems.


{| class="wikitable" border="1"
{| class="wikitable" border="1"
Line 47: Line 47:
| 0x04
| 0x04
| 4
| 4
| Magic 0x30000
| Version, always 0x30000
|-
|-
| 0x08
| 0x08
| 4
| 4
| 0x1000, block size?
| Block size. Always 0x1000.
|-
|-
| 0x0C
| 0x0C
| 4
| 4
| Always 126. Probably device directory capacity. See the [[#Device Directory Capacity]] more information.
| Maximum number of physical files in each physical directory. Always 126. See [[#Device Directory Capacity]] for more information.
|-
|-
| ...
| 0x10
|
 
| The meaning of other fields is unknown
|}
 
== Device Directory Capacity ==
 
A device directory in an extdata (a <code>&lt;SubDirID&gt;</code> directory) seems to have a maximum number of device files it can contain. For SD extdata, this maximum number seems to be hard-coded as 126. For NAND extdata, the number is probably indicated by a field in Quota.dat, which is, again, always 126 as observed. 3DS FS tries to put all device files in the device directory <code>00000000</code> if possible, and only when more than 126 files needed to add, a second device directory <code>00000001</code> and so on are created. However, few extdata have such amount of files to store, so the behavior lacks of use cases to confirm.
 
The number 126 is probably from some kind of other capacity of 128 with <code>&quot;.&quot;</code> and <code>&quot;..&quot;</code> entries reserved. It is theorized that this is to keep a FAT directory table, with 0x20 bytes for each entry, in one 0x1000 cluster. The motivation is unclear.
 
== VSXE File System Metadata ==
 
The inner data of <code>00000000/00000001</code> consists of the following components
 
* VSXE header
* Directory Hash Table
* File Hash Table
* File Allocation Table
* Data region
** Directory Entry Table
** File Entry Table
 
=== VSXE Header ===
 
{| class="wikitable" border="1"
! Offset
! Length
! Description
|-
| 0x00
| 4
| 4
| Magic &quot;VSXE&quot;
| padding
|-
|-
| 0x04
| 0x14
| 4
| Magic 0x30000
|-
| 0x08
| 8
| 8
| File system Information offset (0x138)
| Block count (capacity)
|-
| 0x10
| 8
| Image size in blocks
|-
| 0x18
| 4
| Image block size
|-
|-
| 0x1C
| 0x1C
| 4
| Padding
|-
| 0x20
| 8
| 8
| Unknown
| Number of remaining blocks
|-
|-
| 0x28
| 0x24
| 4
| 4
| 'Action' made on most recently mounted Extdata image
| Current pending operation
{| class="wikitable" border="1"
! Value !! Description
|-
|-
| 0x2C
| 0 || No pending operation.
| 4
| Unknown
|-
|-
| 0x30
| 1 || Physical file deletion is pending
| 4
| D of most recently mounted Extdata image
|-
|-
| 0x34
| 2 || Physical file resize is pending (redundant, see below note)
| 4
| Unknown
|-
|-
| 0x38
| 3 || Physical subdirectory creation is pending
| 0x100
| Mount path, from most recently mounted Extdata image
|-
|-
|
| 4 || Physical subdirectory deletion is pending
 
|}
|
 
| Below is File system Information, which is assumed following the same layout as [[Savegames#SAVE Header
|-
|-
| 0x138
| 0x28
| 4
| Unknown
|-
| 0x13C
| 4
| Data region block size
|-
| 0x140
| 8
| 8
| Directory hash table offset
| Number of available blocks (from offset 0x1C) at the time the above operation was queued (if any).
|-
|-
| 0x148
| 0x30
| 4
| Directory hash table bucket count
|-
| 0x14C
| 4
| Padding
|-
| 0x150
| 8
| 8
| File hash table offset
| File ID that is related to the above operation info. Same as the one in [[Inner_FAT#Filesystem Header|Filesystem Header]]
|-
|-
| 0x158
| 0x38
| 4
| File hash table bucket count
|-
| 0x15C
| 4
| Padding
|-
| 0x160
| 8
| 8
| File allocation table offset
| File size of the above mentioned file. For resize, this is interpreted as the old file size.
|-
|-
| 0x168
| 0x40
| 4
| File allocation table entry count
|-
| 0x16C
| 4
| Padding
|-
| 0x170
| 8
| 8
| Data region offset
| For resize operations, the new file size to apply for the above mentioned file.
|-
| 0x178
| 4
| Data region block count (= File allocation table entry count)
|-
| 0x17C
| 4
| Padding
|-
| 0x180
| 4
| Directory entry table starting block
|-
| 0x184
| 4
| Directory entry table block count
|-
| 0x188
| 4
| Maximum directory count
|-
| 0x18C
| 4
| Padding
|-
| 0x190
| 4
| File entry table starting block
|-
| 0x194
| 4
| File entry table block count
|-
| 0x198
| 4
| Maximum file count
|-
|-
| 0x19C
| 4
| Padding
|}
|}


* All &quot;offsets&quot; are relative to the beginning of VSXE image. All &quot;starting block index&quot; are relative to the beginning of data region.
The quota is implemented as follows:


=== File Allocation Table &amp; Data Region ===
* During initial Extdata creation with [[FS:CreateExtSaveData]], the input size limit is divided by the block size (0x1000). This becomes the block count (capacity).
* Each physical subdirectory (not to be confused with the inner virtual directories in the EXSV filesystem) counts one block towards the block quota.
** Every 126 files, a new physical subdirectory is created.
** This includes the initial physical subdirectory (00000000).
* Each physical subfile (= full DIFF container for a virtual file) counts its file size (rounded up to the quota block size) in blocks towards the block quota.
** This includes all initial physical subfiles, namely <code>icon</code> and the root EXSV filesystem file containing the [[Inner_FAT#Filesystem Header|Filesystem Header]].
** The physical subfile size is taken as the number of blocks corresponding to the physical subfile size rounded up to the block size as specified in the Quota.dat.
* Quota.dat itself is given one block.


These function in the same way as [[Savegames#File Allocation Table|the file allocation in savegames]]. However, the only two &quot;files&quot; allocated in the data region are the directory entry table and file entry table, so the data region is usually pretty small, and the file allocation table is unchanged once created and has no free blocks. Thus, the offset and size of directory / file entry table can be found directly by <code>offset = entry_table_starting block * data_region_block_size + data_region_offset</code> and <code>size = entry_table_block_count * data_region_block_size</code>
Note: Extdata files do not support resizing after their initial creation, because each file is a separate, fixed-size DIFF container. The resize operation is therefore nonfunctional.


=== Directory Hash Table &amp; File Hash Table ===
== Device Directory Capacity ==


These function in the same way as [[Savegames#Directory Hash Table &amp; File Hash Table|those in savegames]]
A device directory in an extdata (a <code>&lt;SubDirID&gt;</code> directory) seems to have a maximum number of device files it can contain. For SD extdata, this maximum number seems to be hard-coded as 126. For NAND extdata, the number is probably indicated by a field in Quota.dat, which is, again, always 126 as observed. 3DS FS tries to put all device files in the device directory <code>00000000</code> if possible, and only when more than 126 files needed to add, a second device directory <code>00000001</code> and so on are created. However, few extdata have such amount of files to store, so the behavior lacks of use cases to confirm.


=== Directory Entry Table ===
The number 126 is probably from some kind of other capacity of 128 with <code>&quot;.&quot;</code> and <code>&quot;..&quot;</code> entries reserved. It is theorized that this is to keep a FAT directory table, with 0x20 bytes for each entry, in one 0x1000 cluster. The motivation is unclear.
 
This functions in the same way as [[Savegames#Directory Entry Table|the one in savegames]]. It lists all virtual directories in this extdata.


=== File Entry Table ===
== VSXE Filesystem ==


This functions in a similar way as [[Savegames#File Entry Table|the one in savegames]]. It lists all virtual files in this extdata. However, the format of a (non-dummy) file entry is a little bit modified:
This is one variant of the [[Inner FAT|FAT filesystem]]. Please refer to its page for the description of the filesystem. In general, device file <code>00000000/00000001</code> contains the metadata of the filesystem, while other device files (except for the Quota file) contains normal sub-files
 
{| class="wikitable" border="1"
! Offset
! Length
! Description
|-
| 0x00
| 4
| Parent directory index in directory entry table
|-
| 0x04
| 16
| ASCII file name
|-
| 0x14
| 4
| Next sibling file index. 0 if this is the last one
|-
| 0x18
| 4
| Padding
|-
| 0x1C
| 4
| <s>First block index in data region</s> '''Always 0x80000000 because unused'''
|-
| 0x20
| 8
| <s>File size</s> '''Unique identifier'''
|-
| 0x28
| 4
| Padding?
|-
| 0x2C
| 4
| Index of the next file in the same hash table bucket. 0 if this is the last one
|}


Each non-dummy file entry corresponds to a device file. The path to the device file is generated by the following computation:
Each non-dummy file entry corresponds to a device file. The path to the device file is generated by the following computation:
Line 303: Line 143:
char extdata_path[...]; // &quot;.../extdata/&lt;ExtdataID-High&gt;/&lt;ExtdataId-Low&gt;&quot;
char extdata_path[...]; // &quot;.../extdata/&lt;ExtdataID-High&gt;/&lt;ExtdataId-Low&gt;&quot;
char device_path[...]; // output path
char device_path[...]; // output path
sprintfdevice_path, &quot;%s/%08x/%08x&quot;, extdata_path, SubDirID, SubFileID);
sprintf(device_path, &quot;%s/%08x/%08x&quot;, extdata_path, SubDirID, SubFileID);
</pre>
</pre>
When mounting extdata, the unique identifier is used to match the ID stored in subfile's [[DISA and DIFF#DIFF header|DIFF header]]. If the ID doesn't match, mounting will fail.
When mounting extdata, the unique identifier is used to match the ID stored in subfile's [[DISA and DIFF#DIFF header|DIFF header]]. If the ID doesn't match, mounting will fail.
Line 435: Line 275:
|  
|  
|-
|-
| ?
| 00000516
| 00000517
| 00000517
| 00000518
| 00000518
Line 539: Line 379:
|-
|-
| ?
| ?
| ?
| 00001132
| 00001131
| 00001131
| Fantasy Life
| Fantasy Life
Line 551: Line 391:
|-
|-
| ?
| ?
| ?
| 000012c8
| 000012ca
| 000012ca
| Mario vs. Donkey Kong: Tipping Stars
| Mario vs. Donkey Kong: Tipping Stars
Line 598: Line 438:
|  
|  
|-
|-
| ?
| 000016C6
| ?
| ?
| 00001678
| 00001678
Line 611: Line 451:
|-
|-
| ?
| ?
| ?
| 0000198e
| 0000198f
| 0000198f
| Animal Crossing: New Leaf - Welcome amiibo
| Animal Crossing: New Leaf - Welcome amiibo
Line 622: Line 462:
|  
|  
|-
|-
| ?
| 00001a2c
| ?
| ?
| 00001a2e
| 00001a2e
Line 709: Line 549:
Empty space is filled with 0xC-long sequences of 00 00 ... 07
Empty space is filled with 0xC-long sequences of 00 00 ... 07


=== Tools ===
== Tools ==
 
* [https://github.com/wwylele/3ds-save-tool 3ds-save-tool] - Extract/verifies extdata
* [https://github.com/wwylele/3ds-save-tool 3ds-save-tool] - Extract/verifies extdata
Retrieved from "https://www.3dbrew.org/wiki/Extdata"