Previously in this series of articles, we’ve started to tear apart the game, beginning with the memory allocator. This time, we’ll take a look at another piece of the puzzle, the code responsible for the data archive.

Where’s all the data?

Back in part 2, we’ve identified TENCHU/DATA.VOL as an archive file in a proprietary format. We could go ahead and reverse-engineer that file format the traditional way, but that’s not how this series rolls. Browsing the strings of MAIN.EXE for any interesting debugging messages, we can find a bunch of them prefixed with Afs and the first 8 bytes of TENCHU/DATA.VOL are AFS_VOL_200.

Coincidence? I think not.

The hunt for AFS

The archive file of Tenchu: Stealth Assassins covers most of the game’s data (besides music, movies and executables), but the game doesn’t directly access it. Instead, it goes through a surprisingly sophisticated indirection mechanism:

There are three data sources implemented in the VFS code:

  • PC, by using libsn functions such as PCopen() and PCread() that only work on a dev kit ;
  • Memory, by accessing additional memory present on a PlayStation equipped with 8 MiB of RAM  ;
  • AFS, by loading files from an AFS archive located on the CD-ROM.

Interestingly, the AFS code doesn’t directly interface with Sony’s libcd library, but goes through an extra layer that provides access to files on the disc through file descriptors.

Retail versions are hard-coded to always initialize the VFS with AFS, since the other two would’ve been useful only during development and none of the other options will work on a standard PlayStation anyway. The memory data source is stubbed out on all versions but the Japanese demo, however the code for the PC data source is still present on all versions.

Currently we only have the AFS archive to work with and we don’t have any data in the correct format to feed the other data sources. We’ll need to extract the AFS archive first to get a hold of the assets.

Delinking afs.o

After some reverse-engineering and annotating, we can identify the following address ranges of MAIN.EXE as part of the AFS archive code:

Section Address range
.rdata 0x80014870 - 0x800149f2
.text 0x8005e950 - 0x8005f227
.sdata 0x80098e78 - 0x80097e7f

Just like the memory allocator, we’ll steal repurpose the game’s original code to run inside our own test programs by delinking it into afs.o, an object file:

$ readelf --wide --file-header afs.o
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           MIPS R3000
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          52 (bytes into file)
  Flags:                             0x0
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         9
  Section header string table index: 8

$ readelf --wide --section-headers --string-dump=.comment afs.o
There are 9 section headers, starting at offset 0x34:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .strtab           STRTAB          00000000 00019c 0003de 00      0   0  1
  [ 2] .symtab           SYMTAB          00000000 00057c 0003c0 10      1  34  4
  [ 3] .rdata            PROGBITS        00000000 000940 000183 00   A  0   0 16
  [ 4] .text             PROGBITS        00000000 000ad0 000928 00  AX  0   0 16
  [ 5] .sdata            PROGBITS        00000000 001400 000008 00  WA  0   0 16
  [ 6] .rel.text         REL             00000000 001408 0002d0 08   I  2   4  4
  [ 7] .comment          PROGBITS        00000000 0016d8 000024 01  MS  0   0  1
  [ 8] .shstrtab         STRTAB          00000000 0016fc 000042 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

String dump of section '.comment':
  [     0]  ghidra-delinker-extension v0.3.0


$ readelf --wide --symbols afs.o

Symbol table '.symtab' contains 60 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS afs.o
     2: 00000000     0 SECTION LOCAL  DEFAULT    3 
     3: 00000000     0 SECTION LOCAL  DEFAULT    4 
     4: 00000000     0 SECTION LOCAL  DEFAULT    5 
     5: 00000000    28 OBJECT  LOCAL  DEFAULT    3 s_AfsOpenVolume:_%s_open_err_80014870
     6: 0000001c    28 OBJECT  LOCAL  DEFAULT    3 s_AfsOpenVolume:_Header_error_8001488c
     7: 00000038    27 OBJECT  LOCAL  DEFAULT    3 s_AfsOpenVolume:_Entry_error_800148a8
     8: 00000054    12 OBJECT  LOCAL  DEFAULT    3 s_AFS_VOL_200_800148c4
     9: 00000064    25 OBJECT  LOCAL  DEFAULT    3 s_AfsGetEntry:_empty_index_800148d4
    10: 00000080    31 OBJECT  LOCAL  DEFAULT    3 s_AfsGetEnty:_memory_not_enough!_800148f0
    11: 000000a0    32 OBJECT  LOCAL  DEFAULT    3 s_AfsGetEntry:_memory_not_enough!_80014910
    12: 000000c0    20 OBJECT  LOCAL  DEFAULT    3 s_Illigal_index_data_80014930
    13: 000000d4    28 OBJECT  LOCAL  DEFAULT    3 s_AfsInit:_not_enough_memory!_80014944
    14: 000000f0    23 OBJECT  LOCAL  DEFAULT    3 s_AfsOpen:_%s_not_found_80014960
    15: 00000108    29 OBJECT  LOCAL  DEFAULT    3 s_AfsOpen:_no_more_handle_[%s]_80014978
    16: 00000128    25 OBJECT  LOCAL  DEFAULT    3 s_AfsClose:_invalid_handle_80014998
    17: 00000144    28 OBJECT  LOCAL  DEFAULT    3 s_AfsFileSize:_invalid_handle_800149b4
    18: 00000160    24 OBJECT  LOCAL  DEFAULT    3 s_AfsRead:_invalid_handle_800149d0
    19: 00000178    11 OBJECT  LOCAL  DEFAULT    3 s_@%d_%.195s_800149e8
    20: 000000b8     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ea08
    21: 0000020c     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005eb5c
    22: 00000348     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ec98
    23: 00000388     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ecd8
    24: 00000410     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ed60
    25: 00000438     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ed88
    26: 0000051c     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005ee6c
    27: 00000584     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005eed4
    28: 00000660     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005efb0
    29: 0000066c     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005efbc
    30: 000006a8     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005eff8
    31: 000006e8     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005f038
    32: 000007a0     4 OBJECT  LOCAL  DEFAULT    4 LAB_8005f0f0
    33: 00000000     5 OBJECT  LOCAL  DEFAULT    5 s_.VOL_80097e78
    34: 00000000   564 FUNC    GLOBAL DEFAULT    4 AfsGetEntry
    35: 00000234   560 FUNC    GLOBAL DEFAULT    4 AfsFind
    36: 00000464   204 FUNC    GLOBAL DEFAULT    4 AfsOpenVolume
    37: 00000530   100 FUNC    GLOBAL DEFAULT    4 AfsInit
    38: 00000594    80 FUNC    GLOBAL DEFAULT    4 FUN_8005eee4
    39: 000005e4   156 FUNC    GLOBAL DEFAULT    4 AfsOpen
    40: 00000680    56 FUNC    GLOBAL DEFAULT    4 AfsClose
    41: 000006b8    64 FUNC    GLOBAL DEFAULT    4 AfsFileSize
    42: 000006f8   196 FUNC    GLOBAL DEFAULT    4 AfsRead
    43: 000007bc   180 FUNC    GLOBAL DEFAULT    4 FUN_8005f10c
    44: 00000870   184 FUNC    GLOBAL DEFAULT    4 AfsCheckMagic
    45: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND AdtMessageBox
    46: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND DiscOpenFile
    47: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND DiscReadFile
    48: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND DiscSeekFile
    49: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND MemoryAllocate
    50: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND MemoryFree
    51: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND memset
    52: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND sprintf
    53: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strcat
    54: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strcmp
    55: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strcpy
    56: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strlen
    57: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strncmp
    58: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND strncpy
    59: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND toupper

$ readelf --wide --relocs afs.o

Relocation section '.rel.text' at offset 0x1408 contains 90 entries:
 Offset     Info    Type                Sym. Value  Symbol's Name
00000038  00000905 R_MIPS_HI16            00000064   s_AfsGetEntry:_empty_index_800148d4
0000003c  00002d04 R_MIPS_26              00000000   AdtMessageBox
00000040  00000906 R_MIPS_LO16            00000064   s_AfsGetEntry:_empty_index_800148d4
00000044  00001504 R_MIPS_26              0000020c   LAB_8005eb5c
00000050  00003104 R_MIPS_26              00000000   MemoryAllocate
00000064  00000a05 R_MIPS_HI16            00000080   s_AfsGetEnty:_memory_not_enough!_800148f0
00000068  00002d04 R_MIPS_26              00000000   AdtMessageBox
0000006c  00000a06 R_MIPS_LO16            00000080   s_AfsGetEnty:_memory_not_enough!_800148f0
00000070  00003204 R_MIPS_26              00000000   MemoryFree
00000078  00001504 R_MIPS_26              0000020c   LAB_8005eb5c
00000090  00003104 R_MIPS_26              00000000   MemoryAllocate
000000a4  00000b05 R_MIPS_HI16            000000a0   s_AfsGetEntry:_memory_not_enough!_80014910
000000a8  00001404 R_MIPS_26              000000b8   LAB_8005ea08
000000ac  00000b06 R_MIPS_LO16            000000a0   s_AfsGetEntry:_memory_not_enough!_80014910
000000b0  00000c05 R_MIPS_HI16            000000c0   s_Illigal_index_data_80014930
000000b4  00000c06 R_MIPS_LO16            000000c0   s_Illigal_index_data_80014930
000000b8  00002d04 R_MIPS_26              00000000   AdtMessageBox
000000c0  00001504 R_MIPS_26              0000020c   LAB_8005eb5c
000000d0  00003004 R_MIPS_26              00000000   DiscSeekFile
000000ec  00002f04 R_MIPS_26              00000000   DiscReadFile
000001ac  00003a04 R_MIPS_26              00000000   strncpy
00000200  00003204 R_MIPS_26              00000000   MemoryFree
00000268  00003a04 R_MIPS_26              00000000   strncpy
00000288  00003b04 R_MIPS_26              00000000   toupper
00000300  00003904 R_MIPS_26              00000000   strncmp
00000354  00003704 R_MIPS_26              00000000   strcpy
00000360  00001305 R_MIPS_HI16            00000178   s_@%d_%.195s_800149e8
00000364  00001306 R_MIPS_LO16            00000178   s_@%d_%.195s_800149e8
0000036c  00003404 R_MIPS_26              00000000   sprintf
00000374  00001704 R_MIPS_26              00000388   LAB_8005ecd8
0000037c  00001604 R_MIPS_26              00000348   LAB_8005ec98
000003c8  00003904 R_MIPS_26              00000000   strncmp
00000418  00001904 R_MIPS_26              00000438   LAB_8005ed88
00000420  00001804 R_MIPS_26              00000410   LAB_8005ed60
00000478  00002504 R_MIPS_26              00000530   AfsInit
00000480  00003804 R_MIPS_26              00000000   strlen
00000498  00003704 R_MIPS_26              00000000   strcpy
000004a4  00002105 R_MIPS_HI16            00000000   s_.VOL_80097e78
000004a8  00003504 R_MIPS_26              00000000   strcat
000004ac  00002106 R_MIPS_LO16            00000000   s_.VOL_80097e78
000004b4  00002e04 R_MIPS_26              00000000   DiscOpenFile
000004c4  00000505 R_MIPS_HI16            00000000   s_AfsOpenVolume:_%s_open_err_80014870
000004c8  00000506 R_MIPS_LO16            00000000   s_AfsOpenVolume:_%s_open_err_80014870
000004cc  00002d04 R_MIPS_26              00000000   AdtMessageBox
000004d4  00001a04 R_MIPS_26              0000051c   LAB_8005ee6c
000004dc  00002c04 R_MIPS_26              00000870   AfsCheckMagic
000004e8  00000605 R_MIPS_HI16            0000001c   s_AfsOpenVolume:_Header_error_8001488c
000004ec  00002d04 R_MIPS_26              00000000   AdtMessageBox
000004f0  00000606 R_MIPS_LO16            0000001c   s_AfsOpenVolume:_Header_error_8001488c
000004f4  00001a04 R_MIPS_26              0000051c   LAB_8005ee6c
000004fc  00002204 R_MIPS_26              00000000   AfsGetEntry
0000050c  00000705 R_MIPS_HI16            00000038   s_AfsOpenVolume:_Entry_error_800148a8
00000510  00002d04 R_MIPS_26              00000000   AdtMessageBox
00000514  00000706 R_MIPS_LO16            00000038   s_AfsOpenVolume:_Entry_error_800148a8
00000550  00003104 R_MIPS_26              00000000   MemoryAllocate
00000560  00000d05 R_MIPS_HI16            000000d4   s_AfsInit:_not_enough_memory!_80014944
00000564  00002d04 R_MIPS_26              00000000   AdtMessageBox
00000568  00000d06 R_MIPS_LO16            000000d4   s_AfsInit:_not_enough_memory!_80014944
0000056c  00001b04 R_MIPS_26              00000584   LAB_8005eed4
0000057c  00003304 R_MIPS_26              00000000   memset
000005b4  00003b04 R_MIPS_26              00000000   toupper
000005fc  00002304 R_MIPS_26              00000234   AfsFind
00000610  00000e05 R_MIPS_HI16            000000f0   s_AfsOpen:_%s_not_found_80014960
00000614  00001c04 R_MIPS_26              00000660   LAB_8005efb0
00000618  00000e06 R_MIPS_LO16            000000f0   s_AfsOpen:_%s_not_found_80014960
0000062c  00001d04 R_MIPS_26              0000066c   LAB_8005efbc
00000658  00000f05 R_MIPS_HI16            00000108   s_AfsOpen:_no_more_handle_[%s]_80014978
0000065c  00000f06 R_MIPS_LO16            00000108   s_AfsOpen:_no_more_handle_[%s]_80014978
00000660  00002d04 R_MIPS_26              00000000   AdtMessageBox
00000690  00001e04 R_MIPS_26              000006a8   LAB_8005eff8
00000698  00001005 R_MIPS_HI16            00000128   s_AfsClose:_invalid_handle_80014998
0000069c  00002d04 R_MIPS_26              00000000   AdtMessageBox
000006a0  00001006 R_MIPS_LO16            00000128   s_AfsClose:_invalid_handle_80014998
000006d0  00001f04 R_MIPS_26              000006e8   LAB_8005f038
000006d8  00001105 R_MIPS_HI16            00000144   s_AfsFileSize:_invalid_handle_800149b4
000006dc  00002d04 R_MIPS_26              00000000   AdtMessageBox
000006e0  00001106 R_MIPS_LO16            00000144   s_AfsFileSize:_invalid_handle_800149b4
00000724  00001205 R_MIPS_HI16            00000160   s_AfsRead:_invalid_handle_800149d0
00000728  00002d04 R_MIPS_26              00000000   AdtMessageBox
0000072c  00001206 R_MIPS_LO16            00000160   s_AfsRead:_invalid_handle_800149d0
00000730  00002004 R_MIPS_26              000007a0   LAB_8005f0f0
0000074c  00003004 R_MIPS_26              00000000   DiscSeekFile
0000077c  00002004 R_MIPS_26              000007a0   LAB_8005f0f0
00000788  00002f04 R_MIPS_26              00000000   DiscReadFile
00000808  00003904 R_MIPS_26              00000000   strncmp
00000888  00003004 R_MIPS_26              00000000   DiscSeekFile
00000898  00002f04 R_MIPS_26              00000000   DiscReadFile
000008a4  00000805 R_MIPS_HI16            00000054   s_AFS_VOL_200_800148c4
000008a8  00003604 R_MIPS_26              00000000   strcmp
000008ac  00000806 R_MIPS_LO16            00000054   s_AFS_VOL_200_800148c4

This time, we have more undefined symbols besides the usual standard C library functions:

  • Memory allocation (MemoryAllocate, MemoryFree) ;
  • Message boxes (AdtMessageBox) ;
  • Disc file functions (DiscOpenFile, DiscReadFile, DiscSeekFile).

We have a couple more functions to shim, but it’s still manageable. We also need the corresponding header file for the C compiler to parse:

#pragma once

#include <stdint.h>

typedef struct {
    uint16_t signature;
    uint16_t type;
    uint32_t location;
    uint32_t size;
    char field1_0xc;
    char field2_0xd;
    char field3_0xe;
    char field4_0xf;
    char name[20];
} afs_entry_t;

typedef struct {
    int32_t used;
    int32_t cursor;
    afs_entry_t* entry;
} afs_file_descriptor_t;

typedef struct {
    disc_file_descriptor_t *discDescriptor;
    char field1_0x4;
    char field2_0x5;
    char field3_0x6;
    char field4_0x7;
    int32_t entriesOffset;
    afs_entry_t* entries;
    int32_t numberEntries;
    int32_t field8_0x14;
    afs_file_descriptor_t* fileDescriptors;
} afs_volume_t;

int AfsGetEntry(afs_volume_t* handle);
afs_entry_t* AfsFind(afs_volume_t* handle, const char* path, unsigned short flags);
int AfsOpenVolume(afs_volume_t* handle, const char* path);
void AfsInit(afs_volume_t* handle);
afs_file_descriptor_t* AfsOpen(afs_volume_t* handle, const char* path);
int AfsClose(afs_file_descriptor_t* fd);
int AfsRead(afs_volume_t* volume, afs_file_descriptor_t* fd, void* buffer, int length);
int AfsCheckMagic(afs_volume_t* handle);

We do not have a definition for disc_file_descriptor_t inside afs.h, as it is part of the disc file subsystem. We will shim it anyways during our investigations in this part.

Puppeteering the game’s code

We have our object file, so we might as well do something interesting with it. But first, we need to provide alternative implementations for the missing symbols before we can start using afs.o:

typedef struct disc_file_descriptor_t disc_file_descriptor_t;

void AdtMessageBox(const char* fmt, ...) {
	va_list args;

	va_start(args, fmt);
	vprintf(fmt, args);
	va_end(args);
}

disc_file_descriptor_t* DiscOpenFile(const char* path) {
	return (disc_file_descriptor_t*) fopen(path, "r");
}

int DiscReadFile(disc_file_descriptor_t* handle, void* buffer, int length) {
	fread(buffer, length, 1, (FILE*) handle);
	return 0;
}

int DiscSeekFile(disc_file_descriptor_t* handle, int offset, int whence) {
	int origin;
	switch (whence) {
		case 0:
			origin = SEEK_SET;
			break;
		case 1:
			origin = SEEK_CUR;
			break;
		case 2:
			origin = SEEK_END;
			break;
		default:
			abort();
	}

	return fseek((FILE*) handle, offset, origin);
}

With that out of the way, it’s time for everyone’s favorite activity: code necromancy.

Proof-of-concept

Let’s start with a single-shot file extractor as a first step, similar to what the game internally does but dumping out the contents to the standard output:

static afs_volume_t vol;

int main(int argc, char* argv[]) {
	if (AfsOpenVolume(&vol, argv[1]) == -1) {
		exit(EXIT_FAILURE);
	}

	afs_file_descriptor_t* fd = AfsOpen(&vol, argv[2]);
	if (fd == 0) {
		exit(EXIT_FAILURE);
	}

	int length = AfsFileSize(&vol, fd);
	void* buffer = MemoryAllocate(length);
	AfsRead(&vol, fd, buffer, length);
	AfsClose(fd);

	write(1, buffer, length);
	MemoryFree(buffer);

	return EXIT_SUCCESS;
}

Let’s build it:

$ mipsel-linux-gnu-gcc -g -static -fno-pic -no-pie -o test-afs.elf test-afs.c afs.o memory.o panic.o
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: afs.o: warning: linking abicalls files with non-abicalls files
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: memory.o: warning: linking abicalls files with non-abicalls files
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: panic.o: warning: linking abicalls files with non-abicalls files

As usual, the toolchain isn’t happy, but it does produce an executable. Looking at MAIN.EXE, we can find the string K:\WORK\CDIMAGE\DEMO\start\card_j.txt embedded within it. Let’s try with it:

$ qemu-mipsel ./test-afs.elf TENCHU/DATA 'K:\WORK\CDIMAGE\DEMO\start\card_j.txt'
[J[hĂ܂.
[J[hĂ܂.
tH[}bgĂ܂\tH[}bg܂?
̃Q[̃f[^\Z[uĂ܂.
[J[h\󂫗eʂ܂.
f[^ǂݍݒł\J[h𔲂Ȃʼn!
f[^ݒł\J[h𔲂Ȃʼn!
ǂݍ݊.
݊.
tH[}bgł\J[h𔲂Ȃʼn!
tH[}bg.
tH[}bgł܂ł.
f[^ǂݍ݂ł܂ł.
f[^݂ł܂ł.
ȑÕf[^Z[uĂ܂\㏑܂?
[J[h`FbNł!
tH[}bgĂ܂.
[J[hĂ܂\J[hĂȂ\f[^Z[uł܂.
[J[h\󂫗eʂ܂\̂܂܂ł̓Z[uł܂.
Z[uł܂񂪂낵ł?
Z[uf[^ǂݍݒł\J[h𔲂Ȃʼn!
Z[uf[^ݒł\J[h𔲂Ȃʼn!
Z[uf[^\ǂݍ݂ł܂ł.
[J[hĂ܂\Z[uł܂񂪂낵ł?
[J[hĂ܂\Z[uł܂񂪂낵ł?
[J[h\󂫗eʂ܂\Z[uł܂񂪂낵ł?
łɃf[^Z[uĂ܂\㏑\Õf[^͖Ȃ܂\Z[u܂?
VɃZ[u܂?
[J[hɃZ[u܂?
㏑܂?
f[^ǂݍ݂ł܂ł\x݂܂?
f[^݂ł܂ł\x݂܂?
̔Cf[^폜܂?
{ɂ낵ł?
폜ł\J[h𔲂Ȃʼn!
폜.
폜ł܂ł.
[J[hĂ܂\ҏWł܂񂪂낵ł?
[hł܂񂪂낵ł?
[J[h\tH[}bgĂ܂\ҏWł܂񂪂낵ł?
̃Q[̖{҃f[^\Z[uĂ܂.
̔Cf[^\Z[uĂ܂.
t@C܂ł.
ׂẴQ[f[^\iō쐬Cf[^ȊOj\㏑܂?

Well, that’s a whole lot of mojibake, but we can discern some structure with the dots and question marks. I could try and find in what code page this is, but rather than wrestling with obsolete character encodings in a foreign language I’ll hazard a guess and try changing the file name to card_e.txt:

$ qemu-mipsel ./test-afs.elf TENCHU/DATA 'K:\WORK\CDIMAGE\DEMO\start\card_e.txt'
Memory Card not found.
Memory Card damaged.
Memory Card not formatted.\Do you want to format it?
There is no saved data\for this game.
Memory Card full.
Reading data.\Don't remove Memory Card!
Writing data.\Don't remove Memory Card!
Finished reading data.
Finished writing data.
Formatting.\Don't remove Memory Card!
Finished formatting.
Formatting failed.
Couldn't read data.
Couldn't write data.
Overwrite\previously saved data?
Checking Memory Card!
Memory Card not formatted.
Memory Card not found.\Insert a Memory Card\for saving data.
Memory Card does not have\enough space for saving data.
Game data can't be saved.\Okay?
Reading data.\Don't remove Memory Card!
Writing data.\Don't remove Memory Card!
Couldn't read saved data.
Memory Card not found.\Game data can't be saved.\Okay?
Memory Card damaged.\Game data can't be saved.\Okay?
Memory Card doesn't have\enough space.\Game data can't be saved.\Okay?
Saving will overwrite\the previous save data.\Save anyway?

Ah, a plain ASCII file that my terminal emulator can render and that I can read. I guess that the first file was also extracted correctly but my terminal couldn’t render whatever 90’s nonsense this was.

The jankiest archive extractor ever

Unfortunately, none of the AFS functions from the game can enumerate the contents of the archive, so we can only extract files this way if we know their full path ahead of time. Therefore, we do need to take a peek at the underlying code and file format to figure this part out.

AFS archives contain a flat index table of files and directories. The first entry is a directory with a straightforward name, but the next ones encode the parent’s entry index in the name.

For example, the following directory tree layout:

  • K:
    • WORK
      • CDIMAGE
        • STAGE
          • YAMIJO
            • TITLE_J.TIM
            • TIM.TPD
            • …
            • IE_3
            • IE_2
          • TEMPLE
            • TITLE_J.TIM
            • TITLE_I.TIM
            • …

Would be encoded like so in the index table:

Index Name Real name Parent index
0 K: K: N/A
1 @0_WORK WORK 0
2 @1_CDIMAGE CDIMAGE 1
3 @2_STAGE STAGE 2
4 @3_YAMIJO YAMIJO 3
5 @4_TITLE_J.TIM TITLE_J.TIM 4
6 @4_TIM.TPD TIM.TPD 4
… … … …
19 @4_IE_3 IE_3 4
18 @4_IE_2 IE_2 4
20 @3_TEMPLE TEMPLE 3
21 @20_TITLE_J.TIM TITLE_J.TIM 20
22 @20_TITLE_I.TIM TITLE_I.TIM 20
… …    

Needless to say, encoding the parent directory’s index number as part of the file name is super weird. I haven’t studied the AFS code in depth, but the mere fact it’s calling sprintf as part of searching for an entry should’ve been a code smell.

Fortunately, we can hack our way around this mess by recomputing the depth of the entries and storing it in an unused field of afs_entry_t and overwriting the buffer for the path as we iterate through the index:

void walk_afs_volume(afs_volume_t* vol, void(*callback)(afs_volume_t* vol, char* name, int isfile)) {
	int depth = 0;
	char path[128] = { 0 };

	strcpy(path, vol->entries[0].name);
	strcat(path, PATH_SEPARATOR);
	vol->entries[0].field1_0xc = 0;

	for (int index = 1; index < vol->numberEntries; index++) {
		afs_entry_t* entry = &vol->entries[index];
		afs_entry_t* parent = &vol->entries[atoi(entry->name + 1)];
		char* component = strchr(entry->name, '_') + 1;

		char* target = path;
		for (int count = parent->field1_0xc; count >= 0; count--) {
			target = strstr(target, PATH_SEPARATOR) + strlen(PATH_SEPARATOR);
		}

		strcpy(target, component);

		if (entry->location == 0xCDCDCDCD) {
			entry->field1_0xc = parent->field1_0xc + 1;
			callback(vol, path, 0);
			strcat(path, PATH_SEPARATOR);
		}
		else {
			callback(vol, path, 1);
		}
	}
}

At this point, writing a brand-new extractor from scratch would’ve probably been a better idea than insisting on piggy-backing the original game code. At any rate, it makes for a decent shakedown of afs.o.

Regardless, with a means to enumerate complete paths we can call AfsOpen with those and extract them like in the previous section. After writing some more code we end up with the unafs.elf utility, with command-line options inspired from tar. It can list the files in a volume:

$ qemu-mipsel ./unafs.elf -tvf TENCHU/DATA
K:
K:\WORK
K:\WORK\CDIMAGE
K:\WORK\CDIMAGE\STAGE
K:\WORK\CDIMAGE\STAGE\YAMIJOU
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_J.TIM
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TIM.TPD
...
K:\WORK\CDIMAGE\STAGE\CAVE\IE_1
K:\WORK\CDIMAGE\STAGE\CAVE\<BA><CB>߰ <81>` STAGE.
K:\WORK\CDIMAGE\STAGE\CAVE\TITLE_E.TIM
...
K:\WORK\CDIMAGE\TRIAL\THEME\MATO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\THEME9.TPD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\NOKI.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO2.TMD

It looks like there’s some data corruption inside the Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen AFS archive…

But more interestingly, it can also extract the whole archive:

$ qemu-mipsel ./unafs.elf -s 1 -xvf TENCHU/DATA
WORK
WORK\CDIMAGE
WORK\CDIMAGE\STAGE
WORK\CDIMAGE\STAGE\YAMIJOU
WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_J.TIM
WORK\CDIMAGE\STAGE\YAMIJOU\TIM.TPD
...
WORK\CDIMAGE\TRIAL\THEME\MATO.TMD
WORK\CDIMAGE\TRIAL\THEME\THEME9.TPD
WORK\CDIMAGE\TRIAL\THEME\UFO.TMD
WORK\CDIMAGE\TRIAL\THEME\NOKI.TMD
WORK\CDIMAGE\TRIAL\THEME\UFO2.TMD

That has to be one of the most convoluted ways to write an asset extractor for a game, but it works.

Interestingly enough, there are files in the archive that are bigger than the available heap memory inside MAIN.EXE. That means we can’t use memory.o delinked from MAIN.EXE and must shim MemoryAllocate and MemoryFree with the C standard memory heap allocator instead.

What’s even more strange is that there’s apparently only one call to AfsOpen (coming from the VFS), which loads files into a buffer allocated dynamically from the memory heap allocated. Either these files aren’t actually used by the game or something else’s going on.

The files for this part can be found here: tenchu1.tar.gz

What about the PC backend?

Going back to the VFS layer, now that we have extracted the assets from the AFS archive we could theoretically use the PC backend…

Patching MAIN.EXE to initialize the VFS layer with the PC backend (while still calling CdInit as the game still expects a CD) is fairly straightforward and DuckStation even supports PCdrv (open Settings > Advanced, check Enable PCDrv under Tweaks/Hacks and set up the path for the PCDrv root directory).

Unfortunately, Tenchu uses backslashes as path separators and relies on case-insensitive paths, owning to the fact it was probably developed on Windows systems. DuckStation doesn’t transform the path in any way and since I’m using Linux, a case-sensitive system with forward slashes as path separators, the game can’t find its files:

$ ls '/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:/WORK/CDIMAGE/IMAGE/IMAGES.ARC' 
'/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:/WORK/CDIMAGE/IMAGE/IMAGES.ARC'
$ duckstation-qt
...
[ 2688.2700] E(HandleSyscall): PCopen: Failed to open '/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:\WORK\CDIMAGE\IMAGE\images.arc'

Even on Windows it won’t work as-is, since the game prefixes any paths with K:\. It’s likely that the PC backend could be made to work again with additional efforts to fix all these issues, but I can’t be bothered to do it right now.

Wait a minute, what’s that doing here?

If we use unafs on the demo version of Rittai Ninja Katsugeki Tenchu, things get really interesting. Backups of files that were probably being modified at the time the demo was finalized, lots of files that the demo probably didn’t use and things that definitively shouldn’t be there:

  • K:\WORK\CDIMAGE\$RES_UP.BAT
  • K:\WORK\CDIMAGE\AFSMAKE.EXE
  • K:\WORK\CDIMAGE\CD.CCS
  • K:\WORK\CDIMAGE\CD.CTI
  • K:\WORK\CDIMAGE\ENDING.EXE
  • K:\WORK\CDIMAGE\GAME.EXE
  • K:\WORK\CDIMAGE\LICENSEJ.DAT
  • K:\WORK\CDIMAGE\PSX.EXE
  • K:\WORK\CDIMAGE\PSX.SYM
  • K:\WORK\CDIMAGE\RESTART.EXE
  • K:\WORK\CDIMAGE\VOLMAKE.BAT

These files (except for LICENSEJ.DAT) can be found here: bonus.tar.gz

We have batch files, CD-ROM generation logs, a bunch of PlayStation executables, debugging symbols for PSX.EXE, the Sony PlayStation license file for Japan (that can be found inside the Psy-Q SDK) and finally AFSMAKE.EXE, a PE i386 console executable for Windows. The mastering of the gold disc for the demo back left some unexpected surprises.

Turns out the Tenchu modding community already knew about these (and also had a working AFS extractor that isn’t scavenged from the corpse of an executable), but it appears this wasn’t publicly documented online before. Still, it’s good I’ve stumbled upon these things before I got any deeper inside my reverse-engineering effort.

Conclusion

We have twisted the original archive code of Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen into a working asset extractor for the AFS file format. We’ve also found some noteworthy artifacts inside the demo version of Rittai Ninja Katsugeki Tenchu that warrant further examination.