Ergo:
I don’t know!
I was shitposting in a Discord server and next thing I know, I’m standing next to a poster in Taipei, showcasing to world-class cybersecurity experts Mad Max-driven development!
I’d better start over from the beginning. I assume you already know about my delinking work; if not, you may imagine Doctor Frankenstein, but with bits instead of gibs.
Right after we left off last time, I casually stumbled upon an invite link to the SURE Discord server whilst idling around the decomp.me Discord server, as one does.
There, I saw a call for papers and this fateful statement was uttered in a Discord channel by a stark raving lunatic yours truly:
Ooh, they have a workshop upcoming and they want non-academics to submit papers? I’ve never had an opportunity to melt down the brains of lots of academics at once before…
… and people actually cheered me for it.
That should’ve been the first hint that something was about to go horribly right.
The call for papers was sent out two months before the deadline and a month had already passed by the time I saw it. I mean, I’m quite adept at burning down computer science literature, how hard can it be to write some?
Incredibly hard.
My initial plans for a long (12 page) paper with quantitative case studies were quickly crushed by time constraints. I cut down the paper down to the bone and then some into a short (6 page) paper with an introduction to the topic and two qualitative case reports, working on it on a month’s worth of spare time.
It took all of the combined might of Gemini and Copilot to massage my first draft into something that vaguely looks like academic vernacular. Because if you haven’t picked up on this by now, my usual style of writing is anything but academic.
So the day arrives, the paper… exists I guess and I submit it, exhausted and deeply unsatisfied by it. Surely, the worst they can say is —
Although we were unable to accept your work for publication, the PC still found your work interesting and worth broader discussion at the workshop. We want to invite you to present your work as a poster at the SURE Workshop on October 13th. We clarify that this work will not appear in proceedings, but we think it can still bring value to authors.
— get over here?
WHAT.
I have received an invitation from a workshop’s program committee, to present a poster at one of the world’s most prestigious academic cybersecurity conferences… on account of a so-called ‘paper’ where I describe the butchering of computer programs into unholy chimeras as a hobby.
This is the kind of world-class conference with a $1810 entry ticket, happening in a convention center whose total area is measured in hectares, attended by professional members of an academic, industrial or state organization, all expenses paid in a five star hotel room with a king bed and a marble bathroom.
You’re attending because you have either managed to land one of the accepted papers (with is by itself an achievement given the exclusive acceptance rate) or you have business to attend alongside the brightest of the field. You don’t just happen to receive an invitation from a program committee to present anything there as a hobbyist. That just, like, doesn’t happen, ever.
There’s only so much I can try to convey how this is utterly impossible in one go, so we’ll set that aside for now.
As for the paper itself, it received two positive and one negative reviews. The main criticisms were:
Turns out academia had done this idea about a decade ago, with the following papers:
I missed that work, partly because when I looked around I only thought about object files and not assembly as a suitable file format.
I will not put this attempt of a paper online. I already thought that it was very weak as a piece of scientific literature before the review. Now that I have the full context on how it fared, bringing it up to a satisfactory standard would take a massive amount of work. Like, a master’s thesis amount of work.
Right, so I’ve basically received an invitation from Hogwarts, school of Witchcraft and Wizardry, to attend their flagship conference. Oh, and it’s in two months at the other side of the planet, whereas I’ve never ventured further away than the next county in my entire adult life… and groaning audibly whenever I do so.
I mean, I don’t have a passport and as a chaotic individual I harbor a deeply-ingrained distrust of bureaucracy. Surely, the authorities will —
| Milestone | Driving license | Identity card | Passport |
|---|---|---|---|
| Requested on | 2025-08-17 | 2025-08-21 | 2025-08-21 |
| In production | 2025-08-19 | 2025-08-26 | 2025-08-27 |
| Arrived on | 2025-09-01 | 2025-09-02 | 2025-09-02 |
— possibly break the speed record of processing on all documents?
That’s less than two weeks, even for my identity card which was renewed for a reason (still valid but requesting the new format) that is explicitly deprioritized. Supposedly, it should’ve taken anywhere from four to six weeks…
Now I’m beginning to believe that this is a fixed point in time, and that failure to attend would unravel the very fabric of the space-time continuum and destroy the entire Universe.
The rest of the required paperwork and supplies were sourced at a leisurely pace (not buying supplies without a conference registration, can’t pay the conference registration fee or hotel deposit without a Visa Premier card, not getting the card until I heard back from my employer…). In hindsight, I could’ve reordered my acquisitions and registrations to better spread the amount of outgoing money.
So I said yes to presenting an academic poster. That means I actually need to have something to hang on a panel. Seeing how bad I’m at writing academic papers, surely me making an academic poster will —
— turn out decent?
The irony that all of the time that I’ve spent doodling… art… during my tertiary education in class, turning out to be a useful skill in an academical context eight years later, is not lost on me. Though I must admit that digital, vector and serious art are all new to me. I also had no idea what an academic poster was supposed to look like, nor did I know what was the meta around them.
It took me six week-ends to wrangle Inkscape into doing something I consider passable (why oh why doesn’t it have geometric constraints?). Yet, despite the amount of work that I’ve put into this, I still hate lots of things about it:
You may think that I’m really hard on myself. It’s the downside of developing an artist’s eye: you spot all of the mistakes in your own work and you are unable to fix them, regardless how many times you’re redoing these parts. Oh, and the longer you stare at your own work, the more mistakes you notice…
I figured I should probably get a business card, as it sounded like something professionals would do. So, I spent another afternoon the week before the departure coming up with this:
My requirements are that I don’t want to order a new print anytime soon, so I carefully designed it so that it contains all of my information that isn’t subject to change, excluding things like job title or employer. I don’t see a point in putting my face on it and although putting my online avatar (an absurdly adorable picture of my cat’s face) was tempting, I’ve decided to put a QR code containing a vCard instead.
I ordered a print of 250 cards on bamboo paper (90% bamboo, 10% cotton). Not because it is ecological, but because it’s unconventional and it gave a look and feel reminiscent of static, analog noise.
As the start of the conference draws near, so does my round-trip to Taiwan. This is my outbound journey:
This is my return trip:
The stage is set, all that remains to do is survive enjoy the ride.
After all, it’s not like my first trip abroad as an adult is to the other side of the planet, or that the last time I flew on an airplane was back when I was a kid, or that I’m heading out there all alone.
No, that’s a brilliant idea.
Triple-checking that I didn’t put my trusty customized N°09 carbon steel Opinel knife in my pocket or cabin luggage, because apparently you can’t really Crocodile Dundee on a commercial airplane anymore, I get all the way to Taipei, unsure of what I’m about to walk into.
This isn’t your typical conference report. I won’t cite here papers or people in particular because I am not a journalist. Instead, I’ll give a meta-report, my general vibe about it.
I’ll set the keynote talks aside, as they were done by engaging and experienced people. They were mostly about giving context regarding a field or a topic, whether for the conference or the workshop. I’ve learned a lot of useful information from them, they were really enlightening and interesting.
I hung out in the SURE workshop all day. It had a lot of attendance (especially for a first edition), never dipping below 40 people and at times the room was nearly completely packed with ~80 people. Overall it was a pleasant experience, quite interesting and insightful, a gentle introduction to the world of academic conferences.
The SURE workshop poster presentation session lasted 90 minutes, concurrently with lunch; I first spent 45 minutes having lunch because I’m French, then I spent 45 minutes in front of my poster, talking non-stop to about 15 people in total.
My poster was an absolute hit, at least within the context of a workshop as it had only three posters, a deliberate choice by the program committee for a focused session. The plastic print was absolutely immaculate and looked even better in person than I hoped for. I’ve had people taking pictures of the poster just because they liked the design that much, before even actually reading it. Some of them showed said pictures to their doctoral advisor, who in turn promptly demanded that their next poster shall look like that.
The main practical problem about my poster during the poster session was that nobody saw the QR codes at the bottom, unless I pointed them out. That whole bottom row was poorly designed in the first place…
I’ve mostly stayed in the software security track, mostly because I was too lazy to actually pick-and-choose what I ought to see.
Imagine sitting all day in a classroom and having a marathon of 15-minute talks on a wide variety of topics given by inexperienced speakers.
I don’t want to be harsh, but most of them were grueling to watch. Stressed-out undergraduates or doctoral students, with thick accents and varying degrees of English fluency, their doctoral advisor most likely in the audience, armed with a set of passable but unremarkable PowerPoint slides…
I have the academic and technical background to understand what was being presented in the software security track, at least at a high level. I don’t have the energy to immerse myself in eighteen or so different topics of research papers in one day, if my brain is already cooked after trying to decode the first set of six talks. It takes a lot of effort to do so and once your mental energy is depleted by that task, you just mentally check out for a couple of minutes.
I actually fell back into my old habits and started doodling on a notepad between my bursts of note-taking, as if I was back in school. Months since I last attempted a drawing, years since I last completed one and by the end of the conference I had a dozen of them.
Somehow, I’ve rekindled a lost hobby of mine. I didn’t see that coming. My drawings were also a hit with the attendees, especially at the bar later on.
It still was worthwhile to attend them, as I’ve learned quite a lot on topics I usually don’t have much experience in. It is however a challenge to stay focused in these conditions all day for those of us like me, who don’t consume caffeinated beverages.
The main track poster session was quite unlike the one from the workshop (and not only because it showcased over 40 posters). By that point, I had watched the Better Posters Generation 1 video and all I will say is that it was spot-on, except that the noise in that cavernous room was also deafening.
The one poster that I found really good and effective was from the people behind WireTap and they also did a demo exhibit… only for me to find out that they were not part of the best poster award competition. I wondered how my poster would’ve fared in this session, but alas it wasn’t meant to be, for mine was confined in a workshop and not on display here.
I didn’t attend, except to eat at the lunch buffet. By that point, I had already developed a really nasty cough, probably because my immune system hadn’t encountered a real challenge in a long time, due to me never adventuring further away than 150 kilometers from home in the past decade. I just chilled, visiting Taipei Civic Plaza and Taipei 101, hanging out with some of the attendees.
Fine, but I’ll do it the lazy way.
I’ve kept notes throughout the conference, but they are basically undecipherable except by me and the Gemini 2.5 Pro chat conversation that contains all of the history regarding my attendance, dating back to me accepting the invitation. I therefore asked it to give a summary, which was structured around the three themes you can read below.
This was arguably the dominant narrative, set by the opening keynote tracing vulnerability research from manual art to autonomous systems driven by competitions like the DARPA Cyber Grand Challenge. Fuzzing remains a primary weapon, now being enhanced with AI/LLMs, Rust-specific awareness, and kernel parameter integration. Defensively, AI is being used for tasks like packer detection, and the fight against obfuscation continues with new synthesis techniques. The field is increasingly characterized by this rapid, automated back-and-forth.
A significant trend is the shift “down the stack,” leveraging hardware features as trust anchors for security. Capabilities, particularly CHERI, were prominent, used for memory safety, sandboxing, and even de-privileging GPU drivers. Trusted Execution Environments (TEEs) and confidential computing were major focus areas, applied to securing I/O for LLMs, defending against firmware attacks, and creating attestable build pipelines. Other hardware interactions, from CPU caches and SSDs to ARM’s MTE and DMA, were also explored as both attack surfaces and defense mechanisms.
Echoing the SURE workshop’s mission to close the “software understanding gap”, there was a clear focus on finding better ways to manage complexity and ensure correctness. This ranged from the highly formal (mechanizing GDPR compliance with temporal logic, creating provably correct parsers) to the practical (improving function signature recovery). A recurring pain point was the “human gap”—developers writing vulnerable code due to fundamental misunderstandings of underlying platforms (like Windows file systems) or security boundaries (like Kubernetes implicit permissions). The SURE panel explicitly debated the need for new, higher-level abstractions beyond source code itself.
There has been three very distinct responses:
As it turns out, by the time you’ve actually solved split relocations, you’ve already created several pieces of a decompiler.
That doesn’t mean that this technique is not useful anymore or not worthy of further study; on the contrary, the fact that I had a growing and enthusiastic user-base in the video game decompilation community was especially of interest.
With the conference finished, I figured that the worst was behind me and that I was home-free.
How wrong I was. The real fun was about to begin.
Come Saturday morning, how hard can it be to check out of a hotel?
Only after explaining to the hotel staff that I was shit out of luck and needed them to suggest a solution, did they proceed to simply use my deposit made during check-in, as it was more than enough to cover my bill.
What did I learn from this?
While being rattled by the experience, even stuck with a non-functional card I still had around NT$4000 and €70 in cash when I left the hotel. It’s not a whole lot, but with a NT$1400 taxi fare to the Taoyuan International Airport as the only foreseeable mandatory expense until my sister grabs me at the Lyon Saint-Exupéry Airport, I figured I was probably fine as long as nothing else went majorly wrong.
I can’t believe I’ve managed to keep it together though this mess. And here I thought that being aloof through immigration with the Taiwan Arrival Card was bad enough…
Remember that short layover at the Hong Kong International airport?
I had originally 100 minutes to perform this layover. The delay stripped 45 minutes off of that, leaving me with less than one hour to speedrun it, which is officially not enough to do it.
Thankfully I didn’t have to find out what would’ve happened should things have gone further awry (more delays, jam and/or issues at the checks…), but that was not a fun experience to do on a full bladder.
Something quite minor, but I’ve made the mistake of eating a yogurt served during breakfast at the end of my HKG-CDG trip. Apparently in the last 15 years since I last ate one, yogurts are no longer welcome in my stomach.
Now that was queasy but manageable… if I hadn’t filled again my water bottle and had to go through another security check after disembarking, drinking yet another half liter of water. It ultimately stayed in throughout the travel area, but not without a fight.
I don’t know!
In case you haven’t noticed, I’m chaos incarnate. I’m the kind of person who ports the NetBSD kernel to the PlayStation 2 while hopped up on anesthetics, after having all of my wisdom teeth pulled out, because I was too smashed to do anything else.
This entire sequence of events, from stumbling upon a call for papers seemingly at random all the way to presenting a poster at ACM CCS 2025 in Taiwan, was so ludicrous that it might as well be taken from a Hollywood movie. This was madness and I have no idea how attempting to decompile a PlayStation video game could’ve possibly snowballed out of control up to that point…
My only explanation is chaos. Pure, unadulterated chaos.
There is so much more to this story than just me performing a live rendition of Crocodile Dundee in the world of software engineering. However, this touches a range of topics that I don’t want to get into on this blog, for a variety of reasons.
Maybe one day I’ll release the Director’s cut of it, but it’s doubtful that I would ever do it publicly. I did share some of these details with the attendees and they were absolutely astounded. As in, the kind of incredible story that you can’t made up, even if you tried.
Well, that trip was entirely self-funded for reasons that I won’t go into and I did keep track of my expenses. So, what does attending ACM CSS 2025 as a hobbyist cost?
| Category | Description | EUR | NT$ | US$ |
|---|---|---|---|---|
| Conference | ACM CCS 2025 Full package registration | -1,552.32 | -1,810.00 | |
| Transport | Air France Airfare, taxes and options | -2,920.17 | ||
| Transport | Air France Airfare insurance | -150.00 | ||
| Supplies | Amazon Travel Supplies (bags, adapter, etc.) | -275.40 | ||
| Supplies | VistaPrint 250 Business Cards Priority shipping | -54.53 | ||
| Personal | Clothes | -2,213.00 | ||
| Personal | Fnac Sony WH-1000MX6 Headphones | -419.99 | ||
| Food & Drinks | Lyon–Saint Exupéry Airport Sandwich | -6.50 | ||
| Currency | Bank of Taiwan EUR to NT$ conversion | -300.00 | +10,401 | |
| Currency | Bank of Taiwan Conversion charge | -30.00 | ||
| Supplies | Taiwan Mobile SIM Card (7 days unlimited data) 5G Short-term Prepaid Card | -800.00 | ||
| Transport | Taxi TPE to Grand Hyatt Taipei | -1,500.00 | ||
| Accommodation | Grand Hyatt Taipei 6 night reservation deposit (authorization hold) | -1,695.97 | -65,000.00 | |
| Food & Drinks | Posino Taipei 777 Wednesday drinks For eight | -4,160.00 | ||
| Entertainment | Taipei 101 Indoor Observatory Deck ticket | -600.00 | ||
| Food & Drinks | Milksha Boba | -150.00 | ||
| Food & Drinks | Milksha Boba - coupon reduction | +20.00 | ||
| Souvenir | Taipei 101 Souvenir Shop Set of tea bags | -580.00 | ||
| Souvenir | Taipei 101 Souvenir Shop Set of three whiskey glasses | -600.00 | ||
| Currency | Grand Hyatt Taipei Advanced cash lending | +2,000.00 | ||
| Transport | Taxi Grand Hyatt Taipei to TPE | -1,400.00 | ||
| Currency | Mega International Commercial Bank NT$ to EUR conversion | +20.00 | -725.00 | |
| Currency | Mega International Commercial Bank Conversion charge | -30.00 | ||
| Accommodation | Grand Hyatt Taipei Remainder from the deposit | +195.26 | +11,908 | |
| Total | -9,372.62 | +1,846 | 0.00 |
Holy shit, that’s just about five digits! This is one expensive mid-life crisis. It’s ahead of schedule too…
The remaining NT$1,846 is cold, hard cash. I’ll keep it as a souvenir, as it is a full set of banknotes and coins, missing only the NT$½ and NT$20 coins as well as the NT$200 and NT$2,000 bills. Besides, it’s only worth about €50 and it’s somewhat hard to convert them here in France (my local bank branch for example does not deal with NT$ cash).
This isn’t a travel blog. Therefore, I’ll spare you all of the various ways a Frenchman, with severely atrophied social skills and a decade out of practice with face-to-face conversational English, bumbles around on the other side of the planet.
But if you really must know my opinions:
I… I think I do know now.
I won’t tell you though, because that would require the Director’s cut of this story and I’ve already decided against publishing it for now. There is however another piece of the puzzle that I can provide at this point. Amid all of that chaos, one small thing still stands out.
A couple of weeks before the conference, I’ve rambled yet again on Hacker News, on a thread about writing science papers. This time, while the comment itself got relatively little traction, I received an email. Not a technical inquiry about my black magic, but one regarding how I was invited to the ACM CCS conference. Here’s a little snippet from it:
This win of yours is frankly, inspirational, and it motivates me to put my work out there, combatting my imposter syndrome.
A fan email. Oh dear Satan, I’m a role model for someone.
Back then, this was a horrifying thought for me. Now, after having completed this extremely out-of-character excursion for me… maybe I can come to terms with things like that. As for the rest, perhaps in time I’ll manage to work through the rest of my personal issues.
Well, I attended the one of the biggest, top-tier academic conferences out there and all I got was a nice thermos flask and a virtual certificate of participation:
I did also come back with the poster and something I never, ever thought I would ever have:
Sure, it’s technically out-of-proceedings regarding the main ACM CCS conference and it’s just a poster, but it still counts I think.
More seriously, that was an experience that I’m glad to have had once. I’ve learned a lot of things, some of them unsuitable for publication on this blog, as well as discussed with a bunch of people I would’ve never met otherwise. I don’t know if I’ll do it again (and not just because of the staggering cost of self-funding this), but the attendees did recommend me to check out an industrial or community conference next time instead.
Now if you excuse me, I’ll need some time to recover from this crazy story… and not only because I’m just done coughing my lungs out. This has been an utterly exhausting endeavor over the past three months; beyond the trip itself, I’ve juggled writing a paper, designing a poster, providing support to the users of ghidra-delinker-extension and a couple of other things alongside a full-time job.
Simply put, I’m spent.
]]>Life has been harrowing lately, but you’re not here for that. Besides, not enough was happening in delinking land to warrant a dedicated post, so I figured I might roll everything up into one update.
My decompilation/reverse-engineering project is still on hiatus, although something did happen.
Back in August 2024, I received an email inquiring about level geometry and freecam. Fast forward a couple of weeks and Sir Rando released a YouTube video with a challenge run for completing the game as Rikimaru without using his sword. You can watch the video here:
The video features in-game B-roll footage with a freecam. I’ve retrofitted this feature into the game with the power of delinking, by replacing the function in charge of the camera with a modified one that gives manual control of it, and keeping the rest of the game’s original code.
Unfortunately, since I stopped at fully delinking Rittai Ninja Katsugeki Tenchu, the two extra levels from the later releases couldn’t benefit from this freecam mod, but the end result still looks nice.
My extension is still snowballing out of control, steadily.
I’ve addressed some tech-debt and released version 0.7.0 back in July 2025. The biggest achievement was refactoring the relocation table from tracking symbol names to tracking target addresses. This solved a whole bunch of problems around symbol name conflicts and symbol name priorization, a feature that picks mangled symbols if available rather than the primary one.
Alongside major improvements in logging and error handling, the extension no longer requires divination (or debugging) to troubleshoot issues with it, meaning that the everyday reverse-engineer has a fighting chance of using it successfully.
The extension seems to have carved a niche for itself in video game decompilation for Windows games. I’ve mentored a fair number of people who reached out to me about using it, with a wide variety of use-cases.
I have second-hand hearsay that my extension is spreading by word of mouth and is being used by people that haven’t reached out to me, but except for download stats (245 downloads for 0.7.0 at the time of writing), I don’t really have any hard data about it.
I know, I know, I really should stop being baited into writing those… Alas, I’ve managed to nail the top comment of a story that stayed at the top of the feed for a while about, fittingly enough, reinventing the wheel.
Two comments from this thread are notable, the first one being:
In my almost 15 years in this community this is the first comment where I don’t have a fucking clue of what is described.
Best Hacker News comment of all times as far as I’m concerned, right after this one:
Pretty awesome work!
That’s from Walter Bright. The Walter Bright.
I guess I know now how it feels when a living legend gives you a thumbs up.
Oh no, I did not unearth this blog and titled this post like so just for that. Remember when I said that ghidra-delinker-extension was snowballing out of control last year?
I had no idea.
You’ll read all about it in my next post… as soon as I get back home from abroad.
]]>For context, as part of my reverse-engineering/decompilation efforts I’ve developed a Ghidra extension that can export relocatable object files. I’ve been working on this tooling over the past two and a half years on my own, but lately something unexpected happened: it started gaining users.
This is a summary of what happened with it over the last couple of months.
Exporting object files is a rather unconventional approach given that assembly files would be a more obvious and natural choice. The key insight here is that object files are essentially containers for arrays of bytes and the container part is mostly unopinionated about its contents. That makes them the real lingua franca of traditional toolchains.
In contrast, there are a lot of assemblers out there, each with their own syntax and quirks. Just for Windows x86 there’s MASM, FASM, NASM, YASM, GAS…
Simply put, object file formats are standardized, interoperable and far less numerous.
Taking this observation into account, I’ve made efforts to architecture my Ghidra extension in a manner that would allow supporting multiple object file formats and instruction sets, even back when all it supported was generating ELF object files for 32-bit little endian MIPS.
Eventually, I’ve added i386 support in order to carry out a case study, but the object file format consideration stayed theoretical until someone contributed a COFF object file exporter. It was very rough around the edges initially, but after spending some weeks fixing all the problems reported by another highly motivated user, it’s fully usable now.
Soon after, someone else stumbled upon my Ghidra extension with an unusual use-case: as part of a video game decompilation project, they wanted to use my extension to delink a program into object files. So far so good, but these object files would not be reused into a program or a library like I usually do.
Instead, these delinked object files would be used as a reference to compare against the output of their decompilation efforts using objdiff, a fancy object file comparator. I’ll admit I was at first a bit skeptical about this, because while my tooling can produce object files that are interchangeable with the original ones, I haven’t designed it to produce equivalent ones.
Fortunately, objdiff has an option for relaxed relocation diffs, where different but equivalent relocations are considered equal. The only actual issue was one of symbol visibility for labels and switch tables, which was accommodated with some new heuristics and options inside the exporters. I didn’t even have to implement section-relative relocations, a feature that I keep postponing.
This incidentally brought me to the decomp.me Discord server, where I’ve discovered that the decompilation community has a term for this kind of tooling: binary splitters, of which splat appears to the most well known. However, my Ghidra extension is quite different than what the community produced, where the focus is more on creating ready-to-use decompilation frameworks.
As an aside, I’ve also received a pull request for an up-to-date Ghidra setup GitHub action. I’ve done the responsible thing, that is quickly scrolling through the GitHub action repository (which is written in a programming language I’m not familiar with) until I said fuck it, I’ll merge it.
After this decision, I started worrying about software supply-chain security for my Ghidra extension. It’s a niche tool with unusual applications, but it’s the kind of stuff that some people with… interesting life stories might use, which in turn might attract attention from other people with… different life stories.
All I’m saying is that while I do not plan on willingly adding undocumented features like a backdoor to my Ghidra extension (not that it has any documentation anyways), that doesn’t mean that Ghidra extensions aren’t a possible threat vector.
It’s very unlikely for my tooling to generate this kind of interest at this point in time, but the XZ Utils backdoor incident has shown that no overworked open-source maintainer is immune to malicious actors.
Yet another person contacted me about using my tooling to delink artifacts that run on IRIX, an Unix operating system by Silicon Graphics, Inc. (or SGI). It seemed straightforward enough, as the ELF and MIPS support was by far the oldest and most mature part of my tooling.
How wrong I was.
The first unanticipated issue was that IRIX ran on big-endian MIPS systems, whereas my tooling was only ever used on little-endian systems up to that point. I had made some nominal accommodations for endianness, but I still ended up refactoring around a quarter of the entire extension in one massive commit anyway.
At least now it should be theoretically possible to delink code for the Nintendo 64 (which is a big-endian system) with my tooling. I’m not going to try though, because I’m far too deep into multiple rabbit holes already.
The second unanticipated issue was position-independent code, which has distinct ABI requirements and specific relocation types (like R_MIPS_GOT16) compared to the kind of MIPS code I usually delink.
Fortunately, retrofitting this into the MIPS analyzers doesn’t appear to require large modifications, as it is fairly similar to the R_MIPS_GPREL16 scheme which is already supported.
Somewhere in the middle of all of this, I figured it was time to try my luck on Hacker News again and this time it took the bait. It stayed for a day on the front page and was even the top item for two hours, as can be seen from these statistics.
In parallel, the GitHub star count ballooned from around 75 stars to over 350 (making it the fourth most starred GitHub repository in the ghidra-extension topic) and the latest version of my Ghidra extension was downloaded 90 times, but for all I know it could’ve been caused by someone integrating my extension as part of a CICD pipeline.
My GitHub account itself was even trending for a bit… near the bottom of the top 25 list in the Java category.
All that ruckus probably doesn’t amount to much. While I’ve put a lot of effort into polishing it, I refuse to believe that my Ghidra extension is so user-friendly that all these people managed to master a very esoteric reverse-engineering technique on their first try without anyone asking for help.
On the surface, all of this sounds good: I’ve created a tool that solves a problem so well that it’s attracting users. While it is unconventional even by the standards of its niche (remember, the French copies nobody and nobody copies the French), somehow a couple of people managed to understand my ramblings and leveraged said tool successfully for their own use-cases.
Under the surface, what was supposed to be a side-quest snowballed into a project that is a massive software engineering challenge. I’ve been meaning to write in detail about it for quite some time, but I keep trashing drafts because I sound like a rambling, stark raving lunatic in them.
Instead, I’ll just state that at this rate it’s simply not sustainable. Mucking around in the internals of ghidra-delinker-extension requires an in-depth technical knowledge about multiple ISAs, object file formats, platforms and toolchains. Making this work flawlessly requires a staggering level of exactness and failure to uphold it will invoke some very exotic undefined behavior.
Currently, my extension supports ELF/MIPS, ELF/x86 and COFF/x86. My grasp on x86, COFF and the MSVC toolchain is tenuous at best and yet it’s most of my known userbase right there.
There are hundreds of valid object file formats and ISAs combinations out there. If people start contributing random combinations to fulfill their use-case, I have no hope of keeping up with that.
Furthermore, despite all my efforts spent on code quality and regression testing, I’m having a hard time keeping the extension from imploding under its own weight. The many, many refactorings I’ve carried out in order to extend the data model and algorithms to fit the ever-increasing scope don’t lend to a sense of stability when the slightest of oversights will break the magic in downright malevolent ways.
Even if one magically handwaves all of these problems away, there are many more hidden behind them. It’s one thing to delink a program, it’s another to reuse the pieces successfully.
The ability to debug the delinked code is an ongoing struggle because my extension does not generate debugging symbols. That would be another dimension for the support matrix on top of ISAs and object file formats, especially if call frame information is involved in order to enable stepping through it. It’s one rabbit hole that I keep restraining myself from jumping into, because that one looks especially cavernous.
My extension also allows delinking to a platform that is different from the original program, in spite of ABIs or common sense. I’ve successfully managed to slightly bend closely-related ABIs together in my experiments, but to really take advantage of this would require some sort of tooling, one that allows mixing and matching ABIs, possibly even ISAs with impunity.
Even if you don’t want to create unholy chimeras, it would be necessary to deal with link-time optimizations in a scalable manner. In that case, toolchains can basically choose to disregard ABIs in the name of size or performance, as long as they can get away with it.
I expect that for the purpose of delinking, this would translate mostly into functions with made-up, nonstandard calling conventions. That sort of tooling would paper over these digressions so that the API boundary presented by the delinked object file respects the ABI standard of its platform.
This one I don’t even know how to begin to solve.
I don’t know.
All I’ve wanted was a way to divide and conquer my decompilation project where I rejected the perfect matching approach used by many others. Instead, I’ve stumbled upon what appears to be an entire unexplored field of computer sciences straight out of some post-apocalyptic cyberpunk universe.
Pages and pages of binary code, ripped out of the carcass of existing programs and stitched together to make new ones, like Frankenstein’s monsters made out of bits instead of gibs. Functions and data backflowing through the one-way toolchain like a big ball of wibbly wobbly, bitsy wimey stuff. Source code and object code blurred past the point where the implications for the GNU General Public License might give lawyers at the Free Software Foundations cirrhosis. Mad Max, but with people scavenging modules out of compiled programs instead of salvaging mechanical parts from cars.
It’s preposterous. Blasphemous, even. Downright heretical. Sorry about that.
What I do know is that I don’t have it in me to do a PhD, let alone several of them, which is what it would take to explore this. Heck, I can barely handle ghidra-delinker-extension by myself in its current form and at times developing it really was like having a strange mood from Dwarf Fortress. This abyss I’ve gazed into is far deeper than I’ve bargained for and it’s giving me vertigo at the moment.
I’m probably going to take a break from developing ghidra-extension-delinker for a couple of weeks or a couple of months. After two and a half years working on giving computer sciences the finger, it’s time for a much-needed break.
I’m not planning on abandoning this at all because it’s far too useful even in its current nascent state, but I need to touch grass with things that are easier on one’s sanity. There’s only so much one feeble human can buck against common wisdom at a time and I’m starting to sound like an insane person on Hacker News.
]]>As mentioned in the introduction of this series of articles, one of the objectives of this project was to make a working, native Linux MIPS port of Tenchu: Stealth Assassins by taking its PlayStation code and shoving it into a Linux process. The idea was to make a Linux port on the cheap as a stepping stone to a fully decompiled and portable version of the game. This “cheap” port concept relied on several assumptions, some of them turned out to be wrong.
I was so preoccupied with whether I could, I didn’t stop to think if I should. It turns out that there’s a price to pay for violating ABIs willy-nilly and I’ve severely underestimated what it would take to achieve this.
In other words, I knew just enough to get into a lot of trouble, but not enough to get out of it.
While qemu-mipsel can emulate a MIPS CPU, it doesn’t support the exact CPU model as found on a PlayStation.
That’s not a problem except for the proprietary geometry transformation engine (or GTE), which is implemented as a coprocessor.
Trying to execute one of its opcodes there will lead to an illegal instruction exception instead of the expected result.
Similarly, qemu-mipsel emulates a user-land process and not a PlayStation, so direct access to hardware registers will not work either.
Luckily, Tenchu: Stealth Assassins is not an optimized game that leverages PlayStation-specific features directly within its code, instead all of that is mediated through the Psy-Q SDK. Therefore, replacing that layer with a drop-in compatible replacement completely eliminates these issues, because the game code itself isn’t strongly tied to the PlayStation hardware. My delinking tooling has already shown this to be possible, even with statically-linked libraries as found in these games.
The fact that the game was written in C on top of the Psy-Q SDK without using any optimization tricks depending on the PlayStation hardware was the one assumption that turned out to be correct. Even if the game accessed the GTE or the hardware directly I can think of multiple ways to deal with that, but I haven’t investigated them due to a lack of need.
As discussed before, there are ways of writing code in a manner that resists delinking when compiling for the MIPS architecture.
This mostly revolves around casting integer constants as pointers, which tends to create instruction patterns (LUI/ORI) that do not match those used for relocations (LUI/ADDIU), meaning we can’t generate a relocation for it.
If you want your MIPS code to be harder to delink, cast raw integer constants as pointers whenever you can.
In particular, do not declare an external variable in your source code and then define the symbol with the linker, either though the linker script or with --defsym SYMBOL=EXPRESSION, like you’re supposed to.
Normally, these can fixed by binary patching the sequence of instructions into a normalized pattern.
However, if the integer constant had its lower 16 bits set to zeroes then the compiler might just load the upper 16 bits with LUI and omit the ORI instruction.
Unless there’s a NOP instruction we can leverage, for example inside an unused branch delay slot, then this can’t be fixed in-place.
I’ve sorted out the remaining apparent delinking issues from the last part, so things like in-game music and cutscenes are working now with the relinked executables on the PlayStation. That being said, not all issues were actually fixed, for references with incorrect instruction patterns to the persistent state and the scratchpad remains.
The fact that the game’s code can survive a round trip through the delinker and still work on the PlayStation doesn’t mean it will work on Linux.
The PlayStation executable code as built by the Psy-Q SDK is what we would call now -fno-pic.
My previous experiments on running that code on Linux relied on building static programs, with position-dependent code and executables, in order to match the ABIs.
As long as I was building console applications, this was a workable solution.
Unfortunately, it is impractical for GUI applications on Linux to be statically linked. The PsyCross SDK depends on SDL, which itself depends on multiple system libraries, including OpenGL. Shared libraries on MIPS mandate position-independent code, which causes the following ABI mismatches:
| No PIC | PIC | |
|---|---|---|
t9 |
Caller-saved | Holds address of callee function |
gp |
Callee-saved | Caller-saved |
This can’t be solved with a simple one-way trampoline.
The gp register must be restored when the function call returns, otherwise we might start using the small data pool of a different shared object than ours, with catastrophic results.
Furthermore, if we divert the execution flow through a thunk, we’d also have to preserve the ra register, otherwise we wouldn’t know where to return to.
Right now I have a kludge of a thunk made up of Makefile magic, shell one-liners and assembly macros.
It saves the ra and gp registers on a shadow stack: it works, but it is not ABI compliant and GDB can’t follow the call chain through it when generating a backtrace.
Fixing this would require implementing a proper thunk mechanism, but doing that correctly and covering all of the edge cases would be a fair amount of work.
Getting PsyCross to cross-compile for Linux MIPS was a bit of a chore, but actually using it was another story.
There are dozens of Psy-Q SDK functions that are missing and that prevents linking an executable with the game’s entire delinked code.
After stubbing all of this (and outright stealing libgs from Psy-Q because PsyCross doesn’t provide it at all), what is implemented doesn’t match Tenchu’s expectations: libcd for example is missing multiple important features that the game relies on in order to load data off the AFS archive.
PsyCross appears to have been originally created for a decompilation project of Tomb Raider: Chronicles before being leveraged for REDRIVER2, the decompilation project for Driver 2: Back on the Streets. It’s a reimplementation of Psy-Q that’s good enough for running a specific game rather than a drop-in compatible replacement for it.
Even though I can get the game executable to link and even technically run on Linux, I do not have a suitable implementation of Psy-Q for this platform to actually play the game. Simply put, PsyCross appears to be way off the mark from what the original game’s code expects and bridging the gap as-is would be a lot of work.
One feature I’ve always postponed for my Ghidra extension is the generation of debugging symbols, mostly because I expect this to be a lot of work even for a partial implementation. So far I’ve accepted a crappy, instruction-based debugging experience inside GDB, but my thunks manage to worsen it significantly.
On top of that, having to run the Frankenstein program through qemu-mipsel adds another layer of pain because QEMU tends to hang and the GDB remote becomes unresponsive when my black magic implodes under its own weight.
This PlayStation-code-on-Linux endeavor requires copious amounts of debugging to sort out issues and I am not willing to entertain this level of development abuse.
With the game’s code on Linux currently stuck inside libcd trying to load data off the AFS archive, it’s executing far enough to demonstrate that an unoptimized PlayStation game can probably be made to run inside a Linux MIPS process, if you abuse it hard enough.
However, the massive number of issues that remains to be solved means that turning this proof-of-concept into a working port or even a useful development environment is currently out of reach.
I’ve already spent two years perfecting the delinking technique as a prelude to this project and right now I’m just not motivated by the idea of going on another Moby-Dick quest to make this work, just to make a point on the Internet. The objective of crafting a Linux MIPS port out of the pieces of the original game’s code as laid out in the introduction is therefore abandoned for the time being.
The lesson learned here is that I need to wield delinking more responsibly: keeping the delinked code on the PlayStation is far easier than trying to shove it by force inside a Linux MIPS process. Given how grueling this investigation was, I’ll probably take a break from this project for now. I still want to reverse-engineer and decompile this game, but not at the cost of the remaining scraps of my sanity.
]]>In this part, I’ve managed to pull off yet another affront to computer science: a rather extreme and unconventional take on binary patching. Specifically, we’ll leverage the delinking technique heralded in this blog to chunk a program into giblets, swap out the parts we want and make the linker mend it all back together in one piece.
This blog post mutilates the desiccated remains of a computer program from 1998 in a most gruesome and unnatural way, all in the name of science. Its contents may be offensive to linker developers, software supply chain specialists or CS101 teachers.
As such, this is your one and only warning to look away before I start the chainsaw.
To understand where this comes from, a quick summary of the history of this project is needed.
As part of my ongoing project to decompile Tenchu: Stealth Assassins, I’ve managed to build a decently annotated Ghidra database of Rittai Ninja Katsugeki Tenchu’s GAME.EXE, the executable for the original JP release.
It’s already a very useful resource for modders of the game, but by itself is only a repository of knowledge, with no practical applications.
In parallel, over the past two years I’ve developed and fine-tuned a rather peculiar reverse-engineering technique known as delinking. What started as a bunch of scrappy Jython scripts eventually evolved into a Ghidra extension that enables one to export a program selection as a working, relocatable object file in two mouse clicks.
These two things combined mean that I can turn GAME.EXE into one or more reusable ELF object files.
In turn, I can relink them to create a new executable and still get something that manages to go in-game without crashing… but I needed something a little extra to write an article on this latest breakthrough.
There are still a bunch of problems to fix inside the Ghidra database and probably also in my Ghidra extension that results in various bugs and crashes, if you tickle the rearranged artifact in the wrong spot.
Nevertheless, I’m still bewildered that this abomination runs at all, let alone that you can stroll around the level, use items and fight enemies as usual.
So, we can delink GAME.EXE back into object files, put these back together with a linker and it mostly works.
This is neat by itself, but it’s not very useful to craft versions of the game with functions and variables just shuffled around in memory.
However, while sitting in a voice chat on the Tenchu Speedrunning Discord server, observing a live stream where bytes were being meticulously patched by hand using Cheat Engine, I got an idea…
What happens if we start changing some of the pieces?
This is something I’ve done previously, but only as a proof-of-concept on a toy program. This time, I’m doing it on a closed-source executable that’s orders of magnitude bigger.
In other words, this is for real.
We can’t just stuff a C source code file at the end of the linker’s invocation alongside the rest of the original object files, as we would run into multiple symbol definition conflicts between the original game’s code and our additional source code. We could manually leave out the parts from the program that we are replacing during object file exportation, but that’s tedious, error-prone and worst of all boring.
What we need is a way to use the original game’s code as-is, but easily override the parts we want with our own code. So after tinkering a bit to find a practical workflow, I ended up with this innocent-looking Makefile:
#!make
-include .env.mak
include settings.mak
define makelibs
$(foreach lib,$(1),-l$(lib))
endef
define makedefsyms
$(foreach sym,$(1),--defsym=$(sym))
endef
define makesrcobjs
$(subst $(1),$(3),$(subst $(2),.o,$(wildcard $(1)/*$(2))))
endef
define makeoverridableobjs
$(filter-out $(sort $(patsubst %,%.o,$(subst $(2)/,$(1)/,$(basename $(wildcard $(2)/*))))),$(wildcard $(1)/*))
endef
all: build/$(EXECUTABLE).exe
clean:
rm -rf build/*.exe build/*.elf build/*.o
build/%.o: src/%.c
$(CROSS_TOOLCHAIN)gcc -fno-pic -mno-abicalls -Wa,-mno-pdr -march=r3000 -mfp32 -I include/ $(CFLAGS) -c $< -o $@
build/%.o: src/%.S
$(CROSS_TOOLCHAIN)as -march=r3000 -mfp32 -mno-pdr -I include/ $(AFLAGS) $< -o $@
build/%.weakened.o: obj/%.o
$(CROSS_TOOLCHAIN)objcopy --weaken-symbols=weak-symbols.txt $< $@
build/$(EXECUTABLE).elf: $(call makesrcobjs,src,.c,build) $(call makesrcobjs,src,.S,build) $(patsubst obj/%.o,build/%.weakened.o,$(call makeoverridableobjs,obj,src))
$(CROSS_TOOLCHAIN)ld -nostdlib -no-pie -static -melf32ltsmip -T $(PSn00bSDK)/libpsn00b/ldscripts/exe.ld -Ttext=$(TEXT_START) $(LDFLAGS) $(call makedefsyms,$(DEFSYMS)) $^ $(call makelibs,$(LIBS)) -o $@
build/$(EXECUTABLE).exe: build/$(EXECUTABLE).elf
ps1-packer $< -o $@
What this Makefile does is create a program from both object files and source code files. To override the original game’s code with our own, we can either provide a source code file that automatically overrides an object file with the same name, or weaken specific symbols from the delinked object files so that our new code is prioritized by the linker.
The layout is as follows:
obj/ ;include/ ;src/ ;build/.The jigs and fixtures of our modding factory are set, all we need to do now is push code through it.
Say we want to modify the debug menu of the game, located within the DoInfoViewProc() function.
With binary patching, we’d have to patch or detour the function, find a place to put our code and data if they don’t fit inline, mend anything that we have moved around or modified, hoping that we didn’t miss anything that would cause crashes or glitches at run-time…
… or we could just write our code, as if nothing is out of the ordinary:
#include "adt.h"
#include "camera.h"
#include "human.h"
#include "infoview.h"
#include "semng.h"
#include "uncategorized.h"
#define NULL (0)
void DoOptions() {
struct TAdtSelect menu[3] = {
{ "Yes", 0 },
{ "No", 1 },
{ NULL, 2 },
};
switch (AdtSelect("Are you cheating son?", menu, 0)) {
case 0: AdtMessageBox("Naughty boy!"); break;
case 1: AdtMessageBox("Good boy!"); break;
default: break;
}
}
void DoSelectItem(unsigned short padTrig) {
int i = SelectedItem;
int direction;
if ((padTrig & 0x2) == 0) {
if ((padTrig & 0x1) == 0) {
return;
}
direction = -1;
} else {
direction = 1;
}
do {
i += direction;
if (i > 0x18) {
i = 0;
} else if (i < 0) {
i = 0x18;
}
} while (((CamState.Owner)->item[i] == 0) && (i != SelectedItem));
SelectedItem = i;
SoundEx(NULL, 0xb);
}
void DoInfoViewProc() {
unsigned short nowPad = GetPad(0);
unsigned short padData = ((CamState.Owner)->pad).data;
unsigned short padTrig = ((CamState.Owner)->pad).trig;
if (fInitialize == 0) {
InitializeInfoView();
}
if ((SystemFlag & 2) && nowPad == 0x0003) {
DoOptions();
}
if ((padData & 0x10) == 0) {
DoSelectItem(padTrig);
}
PutItemList();
PutLifeBar(-0x6e, 0x61, (int)(CamState.Owner)->life, (int)(CamState.Owner)->lifemax);
PutLifeBarS();
PutStrain();
if (((nowPad & 0x0100) == 0) || (SystemFlag & 5)) {
PutMapMode = 0;
} else {
PutMap();
}
PauseProc();
}
Being a PlayStation game from 1998, developed by a newly-created studio staffed with talented but inexperienced people, DoInfoViewProc() does a bunch of things that probably should’ve been divided up into multiple, separate functions.
Since we detour it, our replacement needs to replicate its functionality, hence all the extraneous boilerplate.
Keep in mind that this is a cleaned-up version of the function, the original DoInfoViewProc() is quite a bit more messy and convoluted than this.
Then, we add DoInfoViewProc into weak-symbols.txt, plop object files delinked from GAME.EXE inside obj/, stash our source code inside src/ and let the linker do the work for us:
$ make
mipsel-linux-gnu-gcc -fno-pic -mno-abicalls -Wa,-mno-pdr -march=r3000 -mfp32 -I include/ -std=c11 -Os -Wall -Werror -nostdinc -isystem /home/boricj/Documents/PsyQ/elf/include -c src/cheats.c -o build/cheats.o
mipsel-linux-gnu-gcc -fno-pic -mno-abicalls -Wa,-mno-pdr -march=r3000 -mfp32 -I include/ -std=c11 -Os -Wall -Werror -nostdinc -isystem /home/boricj/Documents/PsyQ/elf/include -c src/libsn.c -o build/libsn.o
mipsel-linux-gnu-gcc -fno-pic -mno-abicalls -Wa,-mno-pdr -march=r3000 -mfp32 -I include/ -std=c11 -Os -Wall -Werror -nostdinc -isystem /home/boricj/Documents/PsyQ/elf/include -c src/__main.c -o build/__main.o
mipsel-linux-gnu-gcc -fno-pic -mno-abicalls -Wa,-mno-pdr -march=r3000 -mfp32 -I include/ -std=c11 -Os -Wall -Werror -nostdinc -isystem /home/boricj/Documents/PsyQ/elf/include -c src/valloc.c -o build/valloc.o
mipsel-linux-gnu-as -march=r3000 -mfp32 -mno-pdr -I include/ -I /home/boricj/Documents/PsyQ/elf/include src/start.S -o build/start.o
mipsel-linux-gnu-objcopy --weaken-symbols=weak-symbols.txt obj/game.o build/game.weakened.o
mipsel-linux-gnu-objcopy --weaken-symbols=weak-symbols.txt obj/psyq.o build/psyq.weakened.o
mipsel-linux-gnu-ld -nostdlib -no-pie -static -melf32ltsmip -T /home/boricj/Documents/PSn00bSDK/libpsn00b/ldscripts/exe.ld -Ttext=0x80010100 -L /home/boricj/Documents/PsyQ/elf/lib --defsym=Scratchpad=0x1f800000 --defsym=PersistentState=0x80010000 --defsym=MemoryPool=0x80200000 --defsym=MemoryDiskSentinel=0x807f0000 build/cheats.o build/libsn.o build/__main.o build/valloc.o build/start.o build/game.weakened.o build/psyq.weakened.o -lcd -lgs -lgpu -lgte -lpad -lmcrd -lcard -lsnd -lspu -letc -lc -lapi -o build/game.elf
ps1-packer build/game.elf -o build/game.exe
ps1-packer by Nicolas "Pixel" Noble
https://github.com/grumpycoders/pcsx-redux/tree/main/tools/ps1-packer/
Input file: build/game.elf
pc: 0x800106b0 gp: 0x00000000 sp: 0x00000000
file size: 700444 -> 194560
File build/game.exe created. All done.
We end up with both an ELF executable file, straight out of our modern toolchain, as well as a PS-EXE file native to the PlayStation.
I could repack the ISO of the game with it, but I want fast iterations so we’ll just start the game, pause execution the moment GAME.EXE gets loaded and inject our version instead:
There are some visible bugs in the modified executable (and it will crash when entering a cutscene), but it mostly works. That program has gone through a meat grinder and got chunked into object files, some bits were taken out and some bits were added… and the linker then stitched it all back together into something that somehow runs.
I feel like a hacksaw-wielding maniac that just got away with mutilating a program in plain sight.
CS101 teachers assume that toolchains are a strict progression of source code through compilers, assemblers and linkers to executable, but actually from a non-linear, unacademic viewpoint it’s more like a big ball of wibbly wobbly, bits-y wimey… stuff.
This is why I say that this modding framework is powered by their tears. If I were pulling this kind of stunts back in university, I would’ve been branded a heretic… or at the very least gave the teaching staff huge migraines.
This is but a single symbol being overridden. What if we wanted to make more ambitious modifications?
Say we want to inject a lot of extra code and data into the game.
The original game’s heap occupies the address range 0x800dc000-0x801fc000, leaving only 16 KiB for the stack at the end of the RAM.
Delinking it will work, but there’s only so much wiggle room left in the program’s memory layout before we collide with the stack.
One way to address it would be to leverage a PlayStation dev kit and its 8 MiB of RAM, so that we can stuff the heap there by telling the linker that the MemoryPool symbol is at address 0x80200000.
This frees 1152 KiB for our modifications, but if our modifications requires a bunch of dynamic allocations then we’re still limited by the original memory allocator’s heap size (and whatever the game left us to play with).
Instead, let’s do something radical and rip it out entirely. We can do so by writing a reimplementation where we can control the heap size, here set to 2 MiB:
#include "valloc.h"
#define NULL (0)
void *memcpy(void *dst, const void *src, unsigned long len);
void *memset(void *dst, int c, unsigned long len);
int sprintf(char *buf, const char *fmt, ...);
#include "adt.h"
extern unsigned char MemoryPool[];
const int MemoryPoolSize = 0x200000;
#define IS_FREE(node) ((node->size & 0x80000000L) == 0)
#define IS_USED(node) ((node->size & 0x80000000L) != 0)
#define MARK_FREE(node) (node->size &= 0x7fffffffL)
#define MARK_USED(node) (node->size |= 0x80000000L)
#define TO_NODE(ptr) (((struct VMhead *)ptr) - 1)
#define FROM_NODE(node) ((void *)(node + 1))
#define GET_NODE_SIZE(node) (node->size & 0x7fffffffL)
#define SET_NODE_SIZE(node, newsize) (node->size = (node->size & 0x80000000L) | newsize)
struct VMhead *virtual_memory_pool;
static void merge_free_nodes(struct VMhead *node) {
while (node->next != NULL && IS_FREE(node->next)) {
node->size += GET_NODE_SIZE(node->next) + sizeof(struct VMhead);
node->next = node->next->next;
}
}
static void split_node(struct VMhead *node, int offset) {
if (GET_NODE_SIZE(node) > (offset + sizeof(struct VMhead))) {
struct VMhead *rest = node->next;
struct VMhead *newNode = (struct VMhead *)(FROM_NODE(node) + offset);
newNode->size = GET_NODE_SIZE(node) - offset - sizeof(struct VMhead);
newNode->next = rest;
SET_NODE_SIZE(node, offset);
node->next = newNode;
merge_free_nodes(node->next);
}
}
void vinit(void *adr, unsigned long size) {
if (adr == NULL) {
adr = &MemoryPool;
}
if (size == 0) {
size = MemoryPoolSize;
}
virtual_memory_pool = adr;
virtual_memory_pool->size = size - sizeof(struct VMhead);
virtual_memory_pool->next = NULL;
}
unsigned long vgetmaxsize(void) {
struct VMhead *node = virtual_memory_pool;
unsigned long maxSize = 0;
while (node != NULL) {
if (IS_FREE(node) && (maxSize < node->size)) {
maxSize = node->size;
}
node = node->next;
}
return maxSize;
}
unsigned long vgetfreesize(void) {
struct VMhead *node = virtual_memory_pool;
unsigned long freeSize = 0;
while (node != NULL) {
if (IS_FREE(node)) {
freeSize += node->size;
}
node = node->next;
}
return freeSize;
}
unsigned long vsize(void *ptr) {
return TO_NODE(ptr)->size & 0x7fffffffL;
}
void *valloc(unsigned long size) {
if (virtual_memory_pool == NULL) {
vinit(NULL, 0);
}
size = (size + 3) & -4;
struct VMhead *node = virtual_memory_pool;
while (node != NULL) {
if (IS_FREE(node) && node->size >= size) {
split_node(node, size);
MARK_USED(node);
return FROM_NODE(node);
}
node = node->next;
}
char buffer[64];
int maxSize = vgetmaxsize();
int freeSize = vgetfreesize();
sprintf(buffer, "OUT OF MEMORY\nREQUEST=%d\nFREE=%d(%d)\n", (int)size, maxSize, freeSize);
SystemOut(buffer);
}
void *vcalloc(unsigned long size, char c) {
void *ptr = valloc(size);
memset(ptr, c, size);
return ptr;
}
void *vrealloc(void *ptr, unsigned int size) {
// FIXME: optimize realloc()
if (size == 0) {
vfree(ptr);
return NULL;
}
void *newptr = valloc(size);
if (ptr != NULL) {
int oldsize = vsize(ptr);
memcpy(newptr, ptr, oldsize < size ? oldsize : size);
vfree(ptr);
}
return newptr;
}
void *vmemoryGC(void *ptr) {
// FIXME: implement vmemoryGC()
return ptr;
}
void vfree(void *ptr) {
if (ptr != NULL) {
struct VMhead *node = TO_NODE(ptr);
if (IS_FREE(node)) {
SystemOut("DOUBLE MEMORY FREE");
}
MARK_FREE(node);
merge_free_nodes(node);
}
}
__attribute((noreturn)) void SystemOut(const char *string) {
AdtMessageBox("*** SYSTEM OUT ***\n\n%s", string);
while (1);
}
This crappy, singly-linked list of memory blocks that wouldn’t look out of place in the 1980s follows the design of the game’s heap allocator implementation, but we could have retrofitted a more modern one, with advanced data structures and better fragmentation characteristics if we wanted.
With our fancy Makefile, if we put this code inside src/valloc.c and the original delinked code inside obj/valloc.o then the source code will be picked over the object file automatically.
There’s no video this time because there are no user-visible changes here, but we’ve just swapped out an entire source file from the original program (that we do not have the source code for by the way) with another, ran make to invoke a couple of standard toolchain programs and it spat out a modified executable that runs.
If this was happening inside a movie, I’d be rolling my eyes at that Hollywood hacking scene.
We’ve leveraged the delinking technique to produce modified versions of an executable, by slicing it into object files and then stuffing some additional source code written by us, before relinking it as a whole again. This proof-of-concept is an interesting alternative to the typical kind of binary patching, where the reverse-engineer needs to contort its modifications within the existing layout of a program.
While the entry ticket is higher because it requires a curated Ghidra database for my delinking tooling to produce usable object files, it allows patching of programs at a far more intrusive scale than traditional techniques can realistically achieve. Constraints from the original program no longer apply when it is dismantled section by section and rebuilt from the ground up.
]]>A while ago, I found a SYM file containing debugging symbols for an early, lost build of the game. That file is a treasure trove of symbol names, data types and function prototypes, far too precious to pass over, which is why I’ve spent a significant amount of time and effort to rescue it.
However, since I eventually want to work on the latest release of the game, this data needs to be version tracked all the way across the entire span of the game’s release timeline to use it there, a multi-step journey all by itself:
PSX.SYM to PSX.SYM.elf, a placeholder program in the shape of the lost build to hold that debugging data inside Ghidra ;PSX.SYM.elf to PSX.EXE, the earliest, non-working artifact we have of the game that is a close match ;PSX.EXE to Rittai Ninja Katsugeki Tenchu’s GAME.EXE, the last version of the game that is similarly architectured into a single monolithic executable ;This third step has been completed, but unlike the previous ones it was just a normal version tracking session that didn’t require any dark magic to pull off, so I didn’t bother writing a dedicated article over that.
This debugging data doesn’t quite cover the entire GAME.EXE file, but I have a mostly-annotated version of the game I can work with for now.
Therefore, while the objective is still to version tracking that data all the way to Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen, I’m putting that goal aside for the time being.
While I was underway with the latest version tracking session, I’ve joined the Tenchu Speedrunning Discord server that serves as the hub of the Tenchu community and announced my reverse-engineering project there.
The Tenchu series hasn’t seen a new release in over 10 years and is a bit of a forgotten cult classic at this point. Also, the speedrunning activity for the original game in particular is at a low ebb compared to a couple of years ago, so that Discord server is a somewhat quiet place these days.
That being said, I did get in touch there with a couple of Tenchu modders that are very interested by the contents of the Ghidra database I’ve built as part of this project. In particular, some lights have been shed on the animation and move set systems of the game (during vocal chat sessions late at night), which is key to creating fully playable versions of enemy and boss characters.
By sharing that data with the Tenchu community, it enables modifications more ambitious in scope than ever before by modders, at least until the decompilation project itself is completed.
It will take a while before we’ll see the results (the modders are still digesting all that new information and experimenting with it as I’m writing this), but at some point some new videos are likely to appear on Teslafane’s YouTube channel, among other places.
I have made several improvements to my Ghidra extension, but I have come to realize that delinking MIPS code in an automated fashion is an absolute, unmitigated nightmare to pull off for several reasons.
While Ghidra has a concept of references, it does not model the relocations I require for delinking. I’m working around this problem with custom-built analyzers that try to identify relocation spots based on the references, but Ghidra’s own automatic analyzers have a tendency to annotate references whenever and however they want, in a manner that confuses and frustrates my tooling and me.
Even with my latest improvements in pattern matching, a fair amount of manual cleanup of references is still required in order for my extension to successfully delink code, especially within large functions or with complex memory access patterns.
The real hair-pulling part, however, comes in with the specifics of the MIPS architecture. In particular, a couple of quirks and features of this instruction set interact together into an absolute mess to pick apart:
I’ve accumulated some nasty test cases inside my test suite, gleaned from the game’s code, that stem from these interactions. This is the kind of gnarly stuff I’m dealing with:
#include <asm/reg.h>
.text
.set noreorder
test:
beq $a0,$zero,1f
lui $v0,%hi(HELLO_WORLD)
lui $v0,%hi(GOODBYE_WORLD)
j 2f
lb $v0,%lo(GOODBYE_WORLD)($v0)
1:
lb $v0,%lo(HELLO_WORLD)($v0)
2:
jr $ra
nop
.data
HELLO_WORLD:
.asciiz "Hello, world!"
GOODBYE_WORLD:
.asciiz "Goodbye, world!"
It might look obvious with the relocation annotations in the assembly code, but remember that all we have to work with inside Ghidra are references on the two lb instructions and a bunch of immediate integer constants.
For this particular case, I had to integrate code block flow analysis into my analyzers in order to untangle the usage of the v0 register.
Unfortunately, that’s not the most messed up example I have. One particularly… creative… instruction pattern present in the game that my Ghidra extension doesn’t handle at the moment looks like this:
LAB_80044468 XREF[1]: 8004443c(j)
80044468 05 00 9e 14 bne a0,s8,LAB_80044480
8004446c 08 80 03 3c _lui v1,0x8008
80044470 01 00 09 24 li t1,0x1
80044474 b8 00 a9 a7 sh t1,local_68(sp)
LAB_80044478 XREF[1]: 80044460(j)
80044478 74 07 80 a7 sh zero,0x774(gp)=>DAT_8008bca8
LAB_8004447c XREF[2]: 800443e4(j), 80044458(j)
8004447c 08 80 03 3c lui v1,0x8008
LAB_80044480 XREF[1]: 80044468(j)
80044480 18 2e 63 24 addiu v1,v1,0x2e18
80044484 01 80 02 3c lui v0,0x8001
80044488 02 00 42 90 lbu v0,offset PersistentState.field0_0x0+2(v0)
8004448c b0 00 a9 97 lhu t1,local_70(sp)
80044490 40 10 02 00 sll v0,v0,0x1
80044494 21 10 43 00 addu v0,v0,v1
80044498 00 00 42 94 lhu v0,0x0(v0)=>DAT_80082e18 = 4Dh M
It’s a bit hard to follow, but basically as far as I can tell the LO16 relocation located inside the addiu instruction at 80044480 has two HI16 counterparts in this snippet:
lui at 8004446c, in the delay slot of the previous bne branch instruction ;lui at 8004447c, right before the target of that same branch instruction.That’s right, the lui instruction targeted by a HI16 relocation has been duplicated.
My best guess is that the assembler figured out it could shave off one instruction from the execution flow by copying the HI16 instruction inside the branch delay slots of the branch instructions targeting it, then shifting the branch targets to one instruction later to skip the original HI16 instruction, which remains for the fallthrough case.
Sep 24, 2024: when I originally wrote this article, I thought that this pattern couldn’t be represented by the R_MIPS_HI16/R_MIPS_LO16 MIPS relocation pair of the ELF file format, which is the object file format I was exporting to.
It turns out that there is a GNU extension for this, as reported by mono21400 on the decomp.me Discord server.
It’s not mentioned inside the SYSTEM V APPLICATION BINARY INTERFACE MIPS® RISC Processor Supplement 3rd Edition document that I was leaning on. The MIPS psABI documentation in general is quite derelict, too.
Furthermore, my data model for synthesized relocations inside my Ghidra extension, which in theory is agnostic of any particular object file format (but in practice is heavily inspired by ELF because that’s what I’m familiar with), also can’t represent this.
A particularly cunning reverse-engineer might observe that patching the bne branch instruction to target one instruction earlier, so that the first lui inside the delay slot would become irrelevant because the second lui that would then get executed in all cases, reduces the HI16/LO16 pattern down to one manageable pair.
It seems to work, but altering the code flow by hand to undo this optimization and normalize the HI16/LO16 relocation pairs is a rather fiddly, intrusive, extreme and error-prone solution… Looks like I’ll have to improvise another crazy contraption to deal with this.
While each improvement of my MIPS analyzers reduces the quantity of unhandled relocation patterns, the remaining ones become harder and harder to address. If there’s a limit to how much bullshit I’m willing to put up with in order to dial in my delinker, this latest issue is getting really, really close to it.
I’ve already repurposed some of its bits and pieces to run on Linux before, but while I can generate an ELF object file of the whole game, it doesn’t mean that it actually works. Or that I can actually link it as a Linux program, for that matter.
On the Linux front, while PsyCross is an obvious seemingly drop-in replacement for the original Psy-Q SDK, it is not currently suitable in practice due to multiple missing functions and even entire libraries used by Tenchu. Even if I were to fill in the blanks and get it to link, splattering lots of PlayStation code inside a Linux MIPS process is bound to create all kinds of havoc, on top of the experimental and temperamental nature of my delinker tooling.
Therefore, a more reasonable stepping stone would be to first delink and relink the game’s code as a PlayStation program using the original Psy-Q SDK. If the game still works with the game’s code and data shuffled around, then the delinking process will be a success and I can go back to running over ABIs like a madman…
But of course it’s not quite that simple.
I do not want to use the original Sony toolchain because it’s thoroughly obsolete and I don’t have an exporter to its proprietary object file format. To work around this, I’m cobbling together an ELF conversion of the Psy-Q SDK out of various tools. This introduces new sources of issues into the mix, but I’m fixing the bugs as I encounter them.
At the moment, I have enough game code delinked and relinked successfully that I can play the briefing screen of a mission correctly and without crashing on a PlayStation emulator, but not much more. Another achievement that’s not worth an article by itself, at least not until I can get the inventory selection screen running properly.
So progress is being made behind the scenes on a bunch of topics, it’s just that nothing’s ready for a write-up at the moment. Having ran out of articles in my buffer for over three weeks now, I figured I might as well write something to document what’s happening.
If it’s not obvious by now, the publication schedule of this blog is definitely not regular anymore. Use the RSS feed if you want to be notified whenever I get around to publish something.
PSX.SYM doesn’t match the PSX.EXE artifact we have on hand.
Undaunted, we’ve created a placeholder program PSX.SYM.elf that matches the layout of PSX.SYM and then loaded the debug data on top of it inside Ghidra.
In this part, we’ll complete the rescue of the debugging data by migrating it to PSX.EXE, a tangible executable.
So, let’s fire up a Version Tracking session with PSX.SYM.elf as the source program and PSX.EXE as the destination program, click on the magic wand icon to run the correlators and…
…there are no matches.
We have a source program with no bytes but metadata and a destination program with bytes but no metadata. With no overlap between the two, Ghidra’s built-in correlators (which work with program bytes, references and symbol names) can’t match anything.
In theory, at this point the only thing we can do with an empty Version Tracking session is to create manual matches by hand, one by one. The source program contains thousands of individual pieces of information, so that’s not a practical option.
First order of business is to improve the quality of the metadata inside PSX.EXE.elf.
While loading PSX.SYM on top of it did bring a lot of information, there are quite a lot of blind spots left in there.
The Psy-Q SDK does not provide any debugging information about its functions and data, but even the game code itself isn’t fully covered.
Before that, running Ghidra’s built-in CreateFunctionsFromSelection.java script on the .text section of PSX.EXE will discover and create the functions that Ghidra’s analyzers missed.
After all, we can’t match functions that do not exist in the destination program.
We have about 450 functions declared inside PSX.SYM, but there are many more functions than that in this executable.
Normally, the built-in CreateFunctionsFromSelection.java script can automatically create functions from dead subroutines, but our placeholder program has no instructions for Ghidra to detect subroutines, so we need another approach.
What we do have for everything are typeless symbols, raw addresses with nothing but names.
If the .text section contains only functions and there are no overlaps or holes, then we can take a symbol and create a function that spans until the next one.
That can be scripted like so:
//Converts a range of symbols to functions, useful for naked symbols inside .text sections.
//@author Jean-Baptiste Boric
//@category Functions
//@keybinding
//@menupath
//@toolbar
import java.util.HashMap;
import java.util.Map;
import ghidra.app.script.GhidraScript;
import ghidra.app.services.*;
import ghidra.app.util.bin.*;
import ghidra.program.model.address.*;
import ghidra.program.model.block.*;
import ghidra.program.model.data.*;
import ghidra.program.model.data.ISF.*;
import ghidra.program.model.lang.*;
import ghidra.program.model.lang.protorules.*;
import ghidra.program.model.listing.*;
import ghidra.program.model.mem.*;
import ghidra.program.model.pcode.*;
import ghidra.program.model.reloc.*;
import ghidra.program.model.scalar.*;
import ghidra.program.model.symbol.*;
import ghidra.program.model.util.*;
import ghidra.util.exception.*;
public class SymbolsToFunctions extends GhidraScript {
public AddressFactory addressFactory;
public FunctionManager functionManager;
public SymbolTable symbolTable;
@Override
public void run() throws Exception {
addressFactory = currentProgram.getAddressFactory();
functionManager = currentProgram.getFunctionManager();
symbolTable = currentProgram.getSymbolTable();
for (AddressRange selectionRange : currentSelection.getAddressRanges()) {
Map<String, AddressSetView> symbolRanges = getSymbolRanges(selectionRange);
for (Map.Entry<String, AddressSetView> symbolRange : symbolRanges.entrySet()) {
String name = symbolRange.getKey();
AddressSetView body = symbolRange.getValue();
Function func = functionManager.getFunctionAt(body.getMinAddress());
if (func == null) {
writer.println(String.format("Creating function %s with body %s", name, body));
functionManager.createFunction(name, body.getMinAddress(), body, SourceType.IMPORTED);
} else {
func.setName(name, SourceType.IMPORTED);
func.setBody(body);
}
}
}
}
private Map<String, AddressSetView> getSymbolRanges(AddressRange addressRange) {
Map<String, AddressSetView> symbolRanges = new HashMap<>();
String previousSymbolName = null;
Address previousSymbolAddress = null;
for (Symbol symbol : symbolTable.getPrimarySymbolIterator(addressFactory.getAddressSet(addressRange.getMinAddress(), addressRange.getMaxAddress()), true)) {
String symbolName = symbol.getName();
Address symbolAddress = symbol.getAddress();
if (previousSymbolAddress != null) {
symbolRanges.put(previousSymbolName, addressFactory.getAddressSet(previousSymbolAddress, symbolAddress.previous()));
}
previousSymbolName = symbolName;
previousSymbolAddress = symbolAddress;
}
if (previousSymbolAddress != null) {
symbolRanges.put(previousSymbolName, addressFactory.getAddressSet(previousSymbolAddress, addressRange.getMaxAddress()));
}
return symbolRanges;
}
}
A fair amount of Psy-Q SDK functions like CdSearchFile use a bunch of private static functions that follow the public function but do not appear in the symbol table.
Since we don’t have the data to reconstruct this, this script will create a big CdSearchFile function instead of one small CdSearchFile function and a bunch of follow-up static functions.
Running this script on a small selection of addresses for testing yields the following output:
SymbolsToFunctions.java> Running...
Creating function cd_tell with body [[8008ae30, 8008ae67] ]
Creating function cd_seek with body [[8008ad8c, 8008ae2f] ]
Creating function cd_read with body [[8008aea0, 8008b02b] ]
Creating function cd_getsize with body [[8008ae68, 8008ae9f] ]
SymbolsToFunctions.java> Finished!
In total, over 640 functions were recovered in this manner. Sure, they don’t have signatures, they don’t have bytes and some of them don’t even have the right size, but we can fix at least the first problem.
Just because the SDK doesn’t provide debugging symbols doesn’t mean we can’t recover information through different means. The Psy-Q headers provide symbol names and type information, at least for the public interfaces. The PlayStation executable loader includes data type archives processed from the SDK header files, which we can apply using this script:
//Applies function signatures held inside data type archives.
//@author Jean-Baptiste Boric
//@category Functions
//@keybinding
//@menupath
//@toolbar
import java.util.Arrays;
import java.util.ArrayList;
import java.util.List;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Pattern;
import ghidra.app.script.GhidraScript;
import ghidra.app.services.*;
import ghidra.app.util.bin.*;
import ghidra.program.model.address.*;
import ghidra.program.model.block.*;
import ghidra.program.model.data.*;
import ghidra.program.model.data.ISF.*;
import ghidra.program.model.lang.*;
import ghidra.program.model.lang.protorules.*;
import ghidra.program.model.listing.*;
import ghidra.program.model.mem.*;
import ghidra.program.model.pcode.*;
import ghidra.program.model.reloc.*;
import ghidra.program.model.scalar.*;
import ghidra.program.model.symbol.*;
import ghidra.program.model.util.*;
import ghidra.util.exception.*;
public class ApplyFunctionSignaturesFromDataTypeArchive extends GhidraScript {
public static final Pattern PATTERN = Pattern.compile(".+/functions/.+");
public AddressFactory addressFactory;
public FunctionManager functionManager;
public SymbolTable symbolTable;
@Override
public void run() throws Exception {
addressFactory = currentProgram.getAddressFactory();
functionManager = currentProgram.getFunctionManager();
symbolTable = currentProgram.getSymbolTable();
DataTypeManager dtm = getDataTypeManager("psyq470");
Map<String, FunctionSignature> signatures = getFunctionSignatures(dtm, PATTERN);
List<Function> functions = new ArrayList<>();
functionManager.getFunctions(currentSelection, true).forEachRemaining(functions::add);
applyFunctionSignatures(functions, signatures);
}
private void applyFunctionSignatures(List<Function> functions, Map<String, FunctionSignature> signatures) throws Exception {
for (Function function : functions) {
FunctionSignature signature = signatures.get(function.getName());
if (signature == null || signature.isEquivalentSignature(function.getSignature())) {
continue;
}
function.setReturnType(signature.getReturnType(), SourceType.IMPORTED);
List<ParameterImpl> parameters = Arrays.asList(signature.getArguments()).stream().map(parameter -> {
try {
return new ParameterImpl(parameter.getName(), parameter.getDataType(), currentProgram);
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}).toList();
function.replaceParameters(parameters, Function.FunctionUpdateType.DYNAMIC_STORAGE_ALL_PARAMS, true, SourceType.IMPORTED);
function.setVarArgs(signature.hasVarArgs());
writer.println(String.format("%s> Processed function %s", function.getEntryPoint(), function.getName()));
}
}
private DataTypeManager getDataTypeManager(String name) {
DataTypeArchiveService dataTypeArchiveService = state.getTool().getService(DataTypeArchiveService.class);
for (DataTypeManager dtm : Arrays.asList(dataTypeArchiveService.getDataTypeManagers())) {
if (dtm.getName().equals(name)) {
return dtm;
}
}
throw new RuntimeException("Couldn't find data type archive " + name);
}
private Map<String, FunctionSignature> getFunctionSignatures(DataTypeManager dtm, Pattern pattern) {
Map<String, FunctionSignature> signatures = new HashMap<>();
List<DataType> dataTypes = new ArrayList<>();
dtm.getAllDataTypes(dataTypes);
for (DataType dataType : dataTypes) {
if (!(dataType instanceof FunctionSignature)) {
continue;
}
FunctionSignature signature = (FunctionSignature) dataType;
if (pattern.matcher(dataType.getDataTypePath().toString()).matches()) {
signatures.put(signature.getName(), signature);
}
}
return signatures;
}
}
This script matches the names of functions to the signatures held inside data type archives and applies the signature to the function if a match is found. We recover over 350 untyped function signatures this way, from the functions we’ve reconstructed from raw labels.
Now that we have cleaned up our placeholder executable as much as possible, it’s time to Version Track it for all its worth. Since the built-in Ghidra correlators can’t match anything, we need to think outside the box. Luckily for us, there are some patterns in the metadata that we can take advantage of.
Assuming the source and destination programs are reasonably close, it is likely that they share global variables and functions with identical sizes, at least to some extent. What is also likely is that they share the same ordering of these global variables and functions, especially within the same original object file. Therefore, if we were to identify an identical pair of spots between the two programs, it’s likely that the data before and after it is also identical.
So we need to identify runs of similarly-sized stuff between our two programs in order to correlate them. I could fork Ghidra and implement new types of correlators cleanly, or I could also just lazily script my way out, which sounds far more expedient for an experimental approach.
Functions are fairly obvious and sizable, so if their sizes have a lot of variation then the patterns should stand out. Since we don’t really know where we’re going with this stuff, we can start with a prototype that merely outputs the best match found:
//Correlates the shapes of functions
//@author Jean-Baptiste Boric
//@category Version Tracking
//@keybinding
//@menupath
//@toolbar
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import ghidra.app.services.*;
import ghidra.app.util.bin.*;
import ghidra.feature.vt.GhidraVersionTrackingScript;
import ghidra.program.model.address.*;
import ghidra.program.model.block.*;
import ghidra.program.model.data.*;
import ghidra.program.model.data.ISF.*;
import ghidra.program.model.lang.*;
import ghidra.program.model.lang.protorules.*;
import ghidra.program.model.listing.*;
import ghidra.program.model.mem.*;
import ghidra.program.model.pcode.*;
import ghidra.program.model.reloc.*;
import ghidra.program.model.scalar.*;
import ghidra.program.model.symbol.*;
import ghidra.program.model.util.*;
import ghidra.util.exception.*;
public class MultipleFunctionShapeCorrelator extends GhidraVersionTrackingScript {
public static final int CONVOLUTION_RANGE = 7;
@Override
public void run() throws Exception {
openVersionTrackingSession("/VT_PSX.SYM.elf_PSX.EXE");
matchFunctions(getFunctionsOf(sourceProgram), getFunctionsOf(destinationProgram));
}
public List<Function> getFunctionsOf(Program program) {
List<Function> functions = new ArrayList<>();
FunctionManager functionManager = program.getFunctionManager();
functionManager.getFunctions(true).forEachRemaining(functions::add);
return functions;
}
public List<Integer> getFunctionSizes(List<Function> functions) {
return functions.stream().map(f -> (int)f.getBody().getNumAddresses()).collect(Collectors.toList());
}
public void matchFunctions(List<Function> sourceFunctions, List<Function> destinationFunctions) {
List<Integer> sourceSizes = getFunctionSizes(sourceFunctions);
List<Integer> destinationSizes = getFunctionSizes(destinationFunctions);
for (int i = 0; i < sourceSizes.size(); i++) {
Function sourceFunction = sourceFunctions.get(i);
int bestMatch = 0;
float bestScore = 0;
float secondBestScore = -1;
for (int j = 0; j < destinationSizes.size(); j++) {
float score = 1;
for (int k = -CONVOLUTION_RANGE; k <= CONVOLUTION_RANGE; k++) {
int a = sourceSizes.get(Math.floorMod(i+k, sourceSizes.size()));
int b = destinationSizes.get(Math.floorMod(j+k, destinationSizes.size()));
score *= 1 - (Math.abs(a - b) / (float)(a + b)) * Math.exp(-Math.abs(k));
}
if (score > bestScore) {
secondBestScore = bestScore;
bestScore = score;
bestMatch = j;
}
}
Function destinationFunction = destinationFunctions.get(bestMatch);
float confidence = (bestScore - secondBestScore) / bestScore;
writer.println(String.format("[%30s; %30s] best score %1.3f second best %1.3f confidence %1.3f", sourceFunction, destinationFunction, bestScore, secondBestScore, confidence));
}
}
}
The script treats the function sizes of both programs as vectors, then tries to find the best pair match by scoring between 0 and 1 how closely the two functions and their neighbors fit together. If we have a score of 1, then the match is perfect: both the functions and their neighbors have the same size. We also compute a confidence score based on the best and second-best scores, the wider the gap between the two the better.
There’s probably an elegant way to modelize this properly and formally, with fancy-pants words like cross-correlation, Gaussian or whatever. Alas, I’m an engineer, not a mathematician, so I’m just winging the scoring with a made-up formula that looks pretty.
As long as the correct match often has the best score, I don’t care if I’m not leveraging this data set to the fullest extent possible.
The script outputs this kind of data on the console:
| Source function | Destination function | Score | Confidence |
|---|---|---|---|
CreateHumanoid |
FUN_800247fc |
0.770 | 0.107 |
KillAllHumanoid |
FUN_800249b8 |
0.913 | 0.249 |
KillHumanoid |
FUN_80024a00 |
0.968 | 0.365 |
ControlAllHumanoid |
FUN_80024af4 |
0.988 | 0.297 |
ControlHumanoid |
FUN_80024b68 |
0.996 | 0.387 |
DefaultActionHumanoid |
FUN_80024de0 |
0.998 | 0.555 |
GetHumanoid |
FUN_800256f0 |
1.000 | 0.457 |
MoveHumanoid |
FUN_80025764 |
1.000 | 0.291 |
Given the lack of mathematical soundness of the formula, we can’t conclude much just by looking at the raw score and confidence numbers. However, we can take a peek at the destination functions and judge if the source function is a match. For this subset, after manual inspection every pair turns out to be correct, so it seems like this empirical method has promising results.
We have the data that pairs functions from the source program to the destination program, but writing text on the output stream doesn’t put it inside Ghidra’s database. Luckily, we can hijack the manual match set present inside any Version Tracking session for our needs.
It’s a bit tricky to add a match in an idempotent manner, to ensure that running the script multiple times doesn’t insert duplicate matches, but not impossible:
private void addMatch(AddressSetView source, AddressSetView destination, float similarity, float confidence, VTAssociationType associationType) {
VTMatchSet manualMatchSet = vtSession.getManualMatchSet();
VTMatchInfo matchInfo = new VTMatchInfo(manualMatchSet);
matchInfo.setSourceAddress(source.getMinAddress());
matchInfo.setDestinationAddress(destination.getMinAddress());
matchInfo.setSourceLength((int) source.getNumAddresses());
matchInfo.setDestinationLength((int) destination.getNumAddresses());
matchInfo.setSimilarityScore(new VTScore(similarity));
matchInfo.setConfidenceScore(new VTScore(Math.pow(10, confidence)));
matchInfo.setAssociationType(associationType);
matchInfo.setTag(null);
if (!manualMatchSet.getMatches().stream().anyMatch(match ->
match.getSourceAddress().equals(matchInfo.getSourceAddress()) &&
match.getDestinationAddress().equals(matchInfo.getDestinationAddress()) &&
(match.getSourceLength() == matchInfo.getSourceLength()) &&
(match.getDestinationLength() == matchInfo.getDestinationLength()) &&
SystemUtilities.isEqual(match.getSimilarityScore(), matchInfo.getSimilarityScore()) &&
SystemUtilities.isEqual(match.getConfidenceScore(), matchInfo.getConfidenceScore()) &&
SystemUtilities.isEqual(match.getAssociation().getType(), matchInfo.getAssociationType())
)) {
manualMatchSet.addMatch(matchInfo);
}
}
After replacing the console logging with a call to this method, we get finally what we were after: a Version Tracking session with a whole bunch of matches.
Not bad for a source program that has no bytes.
In theory, the same operation should be doable with data, but I do not think this is a viable option for now. The function correlator is already having quite a lot of trouble matching objects that are quite distinctive: simply put, I think there’s too little overlapping information between the source and destination program to make it work.
That being said, it’s still possible to match up runs of identical data between PSX.SYM.elf and PSX.EXE by hand, so the debugging information can still be leveraged there.
Just make a selection in the source and destination programs, invoke the Create Manual Match From Tool action and…
…oh come on, I’m not even doing something that’s an offense to Cthulhu right now, I’m just trying to create a manual match with data!
I think this wasn’t implemented before because the built-in correlators probably do a good enough job with bytes and references to not require the creation of manual matches. Too bad my source program doesn’t have bytes or references.
I almost decided to modify Ghidra’s manual match action in anger, but relented after remembering my previous experience maintaining my own Ghidra fork. Instead, I’ll script this one out and maybe worry about making a proper pull request to Ghidra’s source code later:
//Create a manual match for data
//@author Jean-Baptiste Boric
//@category Version Tracking
//@keybinding
//@menupath
//@toolbar
import java.lang.reflect.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import ghidra.app.services.*;
import ghidra.app.util.bin.*;
import ghidra.feature.vt.api.main.*;
import ghidra.feature.vt.gui.plugin.*;
import ghidra.feature.vt.GhidraVersionTrackingScript;
import ghidra.framework.plugintool.*;
import ghidra.program.model.address.*;
import ghidra.program.model.block.*;
import ghidra.program.model.data.*;
import ghidra.program.model.data.ISF.*;
import ghidra.program.model.lang.*;
import ghidra.program.model.lang.protorules.*;
import ghidra.program.model.listing.*;
import ghidra.program.model.mem.*;
import ghidra.program.model.pcode.*;
import ghidra.program.model.reloc.*;
import ghidra.program.model.scalar.*;
import ghidra.program.model.symbol.*;
import ghidra.program.model.util.*;
import ghidra.program.util.*;
import ghidra.util.*;
import ghidra.util.exception.*;
public class CreateManualMatchData extends GhidraVersionTrackingScript {
@Override
public void run() throws Exception {
openVersionTrackingSession("/Miscellanea/VT_PSX.SYM.elf_PSX.EXE (manual matches)");
PluginTool manager = state.getTool();
Plugin subordinate = manager.getManagedPlugins().stream().filter(p -> p.getClass().getSimpleName().equals("VersionTrackingSubordinatePluginX")).findFirst().get();
Field subToolManagerField = subordinate.getClass().getDeclaredField("this$0");
subToolManagerField.setAccessible(true);
VTSubToolManager subToolManager = (VTSubToolManager) subToolManagerField.get(subordinate);
Method sourceMethod = subToolManager.getClass().getDeclaredMethod("getSourceLocation");
Method destinationMethod = subToolManager.getClass().getDeclaredMethod("getDestinationLocation");
sourceMethod.setAccessible(true);
destinationMethod.setAccessible(true);
ProgramLocation sourceLocation = (ProgramLocation) sourceMethod.invoke(subToolManager);
ProgramLocation destinationLocation = (ProgramLocation) destinationMethod.invoke(subToolManager);
CodeUnit sourceCodeUnit = sourceProgram.getListing().getCodeUnitContaining(sourceLocation.getAddress());
CodeUnit destinationCodeUnit = destinationProgram.getListing().getCodeUnitContaining(destinationLocation.getAddress());
AddressSetView source = sourceProgram.getAddressFactory().getAddressSet(sourceCodeUnit.getMinAddress(), sourceCodeUnit.getMaxAddress());
AddressSetView destination = destinationProgram.getAddressFactory().getAddressSet(destinationCodeUnit.getMinAddress(), destinationCodeUnit.getMinAddress().add(source.getNumAddresses() - 1));
if (sourceLocation == null || destinationLocation == null) {
Msg.showInfo(getClass(), null, "Cannot Create Match", "There must be a source and destination selection");
return;
}
addMatch(source, destination, 1.0f, 0.0f, VTAssociationType.DATA);
}
private void addMatch(AddressSetView source, AddressSetView destination, float similarity, float confidence, VTAssociationType associationType) {
VTMatchSet manualMatchSet = vtSession.getManualMatchSet();
VTMatchInfo matchInfo = new VTMatchInfo(manualMatchSet);
matchInfo.setSourceAddress(source.getMinAddress());
matchInfo.setDestinationAddress(destination.getMinAddress());
matchInfo.setSourceLength((int) source.getNumAddresses());
matchInfo.setDestinationLength((int) destination.getNumAddresses());
matchInfo.setSimilarityScore(new VTScore(similarity));
matchInfo.setConfidenceScore(new VTScore(Math.pow(10, confidence)));
matchInfo.setAssociationType(associationType);
matchInfo.setTag(null);
if (!manualMatchSet.getMatches().stream().anyMatch(match ->
match.getSourceAddress().equals(matchInfo.getSourceAddress()) &&
match.getDestinationAddress().equals(matchInfo.getDestinationAddress()) &&
(match.getSourceLength() == matchInfo.getSourceLength()) &&
(match.getDestinationLength() == matchInfo.getDestinationLength()) &&
SystemUtilities.isEqual(match.getAssociation().getType(), matchInfo.getAssociationType())
)) {
manualMatchSet.addMatch(matchInfo);
}
}
}
By the sheer amount of Java reflection going on in this script, you can probably tell I was getting rather pissed off trying to accomplish this task. I’m scripting the crap out of Ghidra here, that’s not how you’re supposed to do it.
Anyways, this script works similarly to the Create Manual Match From Tool action.
Click a spot on the source and destination programs, it will create a match from the source code unit to an equally-sized region on the destination.
The home-built function correlator worked very well for game code covered by debugging symbols, but it struggles in the following situations:
Creating manual matches for the misidentified functions and the bulk of the data sections took a bit of time, but at the end the destination program is fairly thoroughly annotated, at least for the game and public SDK parts. The fact that the source and destination programs were very close made the process reasonably straightforward, but some of the matches did require a lot of inferring to figure out.
Had I attempted to version track directly onto a more distantly related artifact, it is likely that the source program being a placeholder with no initialized bytes or references would’ve made identifying matches extremely difficult.
Despite the built-in Ghidra correlators being unable to work with PSX.SYM.elf, our placeholder program created in the last part, we’ve managed to make matches inside Version Tracking against PSX.EXE through out-of-the-box thinking and generous (ab)use of scripting, rescuing the valuable debugging data from a lost artifact onto a real program.
PSX.EXE and PSX.SYM, an early build of Tenchu’s main executable with its accompanying debugging information file.
Found hidden inside the AFS archive of the demo version of Rittai Ninja Katsugeki Tenchu, PSX.EXE raises a number of questions, particularly when it was created.
For reference, the timeline looks like this:
K:\WORK\CDIMAGE\CD.CCS file inside its AFS archive contains the date September 5th, 1997 ;Do not quote me on the dates, I did not rely on authoritative sources for them.
Looking at the artifact itself, PSX.EXE does contain strings for all 8 levels plus the training level as well as a SCCS string for bios.c dated March 28th, 1997, which means it could’ve been built inside a time span of approximately half a year until the demo release date.
It seems to fit the way the original JP releases were architectured, with one big executable instead of four separate executables as found on the NA release onward.
From the NA release of Tenchu: Stealth Assassins onwards, a 4-byte binary-coded decimal date is stored at address 0x80010000 during initialization, but unfortunately that doesn’t apply to earlier versions.
It’s likely that it is the last built executable before the development team switched to the name GAME.EXE and it was simply left there, forgotten.
Merely substituting either GAME.EXE or PAPX_900.29 on the demo version of Rittai Ninja Katsugeki Tenchu with PSX.EXE doesn’t seem to yield a working game.
At any rate, the really juicy bit isn’t PSX.EXE itself, but PSX.SYM.
PSX.SYM contains debugging information in a proprietary MND format, specific to the Psy-Q toolchain.
There’s some reverse-engineered documentation, a decompilation for DUMPSYM.EXE and some tools to manipulate them.
The ghidra_psx_ldr extension I’ve been using for importing PS-EXE files used to support SYM files, but that feature was removed in 2020.
Anyway, since I have my own ideas about my workflow I started writing a Java script (note the space) for Ghidra to process this file and get all that sweet metadata inside the program database, but it soon became apparent that things weren’t that simple…
Unfortunately, the addresses inside PSX.SYM do not match the contents of PSX.EXE.
This means that this SYM file was for another, most likely earlier version of PSX.EXE, which we don’t have.
There’s still tons of information in there, like file names, data types, function definitions and symbol names, we just can’t directly apply it to any artifact on hand.
However, the order of symbols inside PSX.SYM does appear to match PSX.EXE, at least to some extent, so it can’t be that far off.
That’s way too valuable to pass up. Therefore, the new plan currently looks something like this:
PSX.SYM and all its information as a program into Ghidra ;PSX.SYM to PSX.EXE, to get that data onto an executable we have ;PSX.EXE to Rittai Ninja Katsugeki Tenchu’s GAME.EXE ;After this, the valuable debugging data from PSX.SYM will have been transferred to an executable that we can actually play with, both literally and figuratively.
From there, assuming there wasn’t too much transmission loss during all these migrations, I can use the annotated Rittai Ninja Katsugeki Tenchu’s GAME.EXE as a new starting point for my reverse-engineering effort.
But how do we load the debugging symbols into Ghidra in the first place if we do not have a program to begin with?
Okay, so we have a SYM file but not its PS-EXE executable. That’s like having an egg’s shell and membrane but not its yolk and albumen. Ghidra being a hungry beast, it can’t feast on an egg (debugging symbols) devoid of its substance (executable)… but maybe if we pad the inside with tofu it won’t know any better?
Sketchy metaphors aside, without a program to load we can’t process debugging information inside Ghidra. However, if we can scaffold a placeholder program, one that matches the shape if not the contents of the missing executable, then we can load it and then lay that debugging information on top of this fake program.
For our placeholder executable, we’ll need a header, a section table and the section’s bytes. We’ll pad the bytes with dummy data, but we still need the sections themselves. Fortunately, the SYM file contains symbols that mark the start, end and size of each section:
$ dumpsym ~/Games/'Rittai Ninja Katsugeki - Tenchu (Japan) (Demo)'/K:/WORK/CDIMAGE/PSX.SYM | grep -E '__[a-z]+(_org|_size)' | cut -f2,4 -d' ' | sort
$00000000 __ctors_size
$00000000 __dtors_size
$0000018e __sbss_size
$0000069c __sdata_size
$00003762 __rdata_size
$0000ca22 __data_size
$000777b0 __text_size
$0015aad8 __bss_size
$80010100 __ctors_org
$80010100 __ctors_orgend
$80010100 __dtors_org
$80010100 __dtors_orgend
$80010100 __rdata_org
$80013862 __rdata_orgend
$8001387c __text_org
$8008b02c __data_org
$8008b02c __text_orgend
$80097a4e __data_orgend
$80097a50 __sdata_org
$800980ec __sdata_orgend
$80098118 __sbss_org
$800982a6 __sbss_orgend
$800982c8 __bss_org
$801f2da0 __bss_orgend
With this, we can recreate the memory map for the sections of our placeholder:
| Section | Start address | End address | Size | Padded size |
|---|---|---|---|---|
.rdata |
0x80010100 |
0x80013862 |
0x3762 |
0x377c |
.text |
0x8001387c |
0x8008b02c |
0x777b0 |
0x777b0 |
.data |
0x8008b02c |
0x80097a4e |
0xca22 |
0xca24 |
.sdata |
0x80097a50 |
0x800980ec |
0x69c |
0x6c8 |
.sbss |
0x80098118 |
0x800982a6 |
0x18e |
0x1b0 |
.bss |
0x800982c8 |
0x801f2da0 |
0x15aad8 |
0x15aad8 |
The size of the sections will be padded to the start of the next section because there are symbols wedged in between the sections, at least according to this file.
Once more, the toolchains in the 90s didn’t give a damn…
One extra bit of information we can observe is that the start symbol is located at address 0x80098098, which will be our entrypoint.
After that, we can then repurpose some Jython code from my reverse-engineering series of articles to hand-craft the placeholder executable.
Ghidra embeds a Jython standalone interpreter at Ghidra/Features/Python/lib/jython-standalone-2.7.3.jar, which we can use to run this script:
import struct
from jarray import zeros
ELFCLASS32 = 1
ELFDATA2LSB = 1
EV_CURRENT = 1
ELFOSABI_SYSV = 0
ELF_ET_REL = 1
ELF_ET_EXEC = 2
ELF_EM_MIPS = 8
ELF_EV_CURRENT = 1
ELF32LE_HDR = "<4c12BHHIIIIIHHHHHH"
ELF32LE_HDR_SIZE = struct.calcsize(ELF32LE_HDR)
SHT_NULL = 0
SHT_PROGBITS = 1
SHT_SYMTAB = 2
SHT_STRTAB = 3
SHT_NOBITS = 8
SHF_WRITE = 0x1
SHF_ALLOC = 0x2
SHF_EXECINSTR = 0x4
ELF32LE_SHDR = "<IIIIIIIIII"
ELF32LE_SHDR_SIZE = struct.calcsize(ELF32LE_SHDR)
# Craft string tables
def craft_string_table(strings):
data = ""
offsets = dict()
for string in strings:
offsets[string] = len(data)
data += (string + "\0").encode("ascii")
return data, offsets
_shstrtab_bytes, shstrtab = craft_string_table(["", ".shstrtab", ".rdata", ".text", ".data", ".sdata", ".sbss", ".bss"])
# Name Type Flags Address Bytes Link Info Alignment Entry size
sections = [
("", SHT_NULL, 0, 0, "", 0, 0, 0, 0),
(".shstrtab", SHT_STRTAB, 0, 0, _shstrtab_bytes, 0, 0, 0, 0),
(".rdata", SHT_NOBITS, SHF_ALLOC, 0x80010100, 0x377c, 0, 0, 0, 0),
(".text", SHT_NOBITS, SHF_ALLOC|SHF_EXECINSTR, 0x8001387c, 0x777b0, 0, 0, 0, 0),
(".data", SHT_NOBITS, SHF_WRITE|SHF_ALLOC, 0x8008b02c, 0xca24, 0, 0, 0, 0),
(".sdata", SHT_NOBITS, SHF_WRITE|SHF_ALLOC, 0x80097a50, 0x6c8, 0, 0, 0, 0),
(".sbss", SHT_NOBITS, SHF_WRITE|SHF_ALLOC, 0x80098118, 0x1b0, 0, 0, 0, 0),
(".bss", SHT_NOBITS, SHF_WRITE|SHF_ALLOC, 0x800982c8, 0x15aad8, 0, 0, 0, 0),
]
file_offset = ELF32LE_HDR_SIZE + ELF32LE_SHDR_SIZE * len(sections)
elf_section_headers_bytes = ""
for section in sections:
length = section[4] if type(section[4]) == int else len(section[4])
elf_section_headers_bytes += struct.pack(ELF32LE_SHDR,
shstrtab[section[0]], section[1], section[2], section[3], file_offset, length, section[5], section[6], section[7], section[8]
)
file_offset += len(section[4]) if type(section[4]) != int else 0
elf_header_bytes = struct.pack(ELF32LE_HDR,
'\x7f', 'E', 'L', 'F', ELFCLASS32, ELFDATA2LSB, EV_CURRENT, ELFOSABI_SYSV, 0, 0, 0, 0, 0, 0, 0, 0,
ELF_ET_EXEC, ELF_EM_MIPS, ELF_EV_CURRENT, 0x80098098, 0, ELF32LE_HDR_SIZE, 0x1000, ELF32LE_HDR_SIZE, 0, 0, ELF32LE_SHDR_SIZE, len(sections), 1
)
with open("PSX.SYM.elf", "w") as fp:
fp.write(elf_header_bytes)
fp.write(elf_section_headers_bytes)
for section in sections:
if type(section[4]) != int:
fp.write(bytearray(section[4]))
Since PSX.SYM.elf is a real ELF file that happens to be a fake executable, we can inspect it with the usual tools:
$ readelf --wide --file-header PSX.SYM.elf
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: MIPS R3000
Version: 0x1
Entry point address: 0x80098098
Start of program headers: 0 (bytes into file)
Start of section headers: 52 (bytes into file)
Flags: 0x1000, o32, mips1
Size of this header: 52 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 40 (bytes)
Number of section headers: 8
Section header string table index: 1
$ readelf --wide --section-headers PSX.SYM.elf
There are 8 section headers, starting at offset 0x34:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000174 000000 00 0 0 0
[ 1] .shstrtab STRTAB 00000000 000174 000030 00 0 0 0
[ 2] .rdata NOBITS 80010100 0001a4 003762 00 A 0 0 0
[ 3] .text NOBITS 8001387c 0001a4 0777b0 00 AX 0 0 0
[ 4] .data NOBITS 8008b02c 0001a4 00ca22 00 WA 0 0 0
[ 5] .sdata NOBITS 80097a50 0001a4 00069c 00 WA 0 0 0
[ 6] .sbss NOBITS 80098118 0001a4 00018e 00 WA 0 0 0
[ 7] .bss NOBITS 800982c8 0001a4 15aad8 00 WA 0 0 0
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)
It’s a whole lot of nothing: this executable doesn’t have segments for the loader to process or even a single byte of initialized memory to its name. QEMU won’t even let us try to run it:
$ chmod +x PSX.SYM.elf
$ qemu-mipsel PSX.SYM.elf
qemu-mipsel: PSX.SYM.elf: Invalid ELF image for this architecture
Despite all of that, Ghidra will happily import this so-called executable. After all, all we need it to do is to act as a stand-in, so that the debugging data has a place to live.
Now that we have a placeholder executable, all we need to do is to layer that debugging data on top of this scaffolding. It’s easier said than done because this is a deceptively tricky task:
I also have some additional requirements that piles another level of difficulty on top of this:
After banging my head on the problem for quite a while, I’ve managed to write a battle-scarred but working Ghidra importation script for SYM files, which has since been upstreamed to the ghidra_psx_ldr extension.
We have symbols:
We have data:
We even have functions:
Sure, we haven’t got a byte for any of this stuff: PSX.SYM.elf is just a placeholder after all.
However, if we can port all that metadata onto an executable we do have, then we’ll have a fully annotated artifact with real bytes…
but that is a story for another part.
We’ve discovered that the debugging symbols file PSX.SYM doesn’t match PSX.EXE, meaning that we have a debugging symbols file and no program to use it on.
Undaunted, we’ve created a fake program PSX.SYM.elf that matches the outline of PSX.SYM and then wrote a script to import all this information inside Ghidra, as if we had the real program on hand.
AFSMAKE.EXE, presumably used to create the proprietary AFS archive files for this game.
AFSMAKE.EXE is a 32-bit Windows PE program.
To run it on a x86_64 Debian system, Wine needs to be installed first:
$ sudo dpkg --add-architecture i386
$ sudo apt-get update
$ sudo apt-get install \
wine \
wine32
With that out of the way, it’s time to check out the loot:
$ wine AFSMAKE.EXE
option:
-p : pack
-x : extract
-l : list
$ wine AFSMAKE.EXE -l
Afs file list
option: <file>
$ wine AFSMAKE.EXE -p
AFS Volume Maker Version 2.00
optins:
<volume file name> <file>
$ wine AFSMAKE.EXE -x
optins:
<vol name> <filename> <destination>
Inconsistent formatting, capitalization and typos: telltale signs of a homemade tool. Is the lack of veneer superficial or is this program jank through and through? Only one way to find out.
Let’s try a simple task, listing the files of the demo archive:
$ wine AFSMAKE.EXE -l TENCHU/DATA.VOL
âtâ@âCâïé¬éPé┬éαéáéΦé▄é╣é±
Curses! Obsolete 90’s character encoding strikes again and this time there aren’t any conveniently located English translations nearby. We’ll have to figure this one out.
Since we are dealing with a Windows program from Japan, it’s a fair bet that this is encoded in code page 932, also known as Shift-JIS. I’m too lazy to figure out how to tell Wine and my terminal emulator how to interpret this stuff, so we’ll use Python to do the conversion for us:
$ wine AFSMAKE.EXE -l TENCHU/DATA.VOL > mojibake.txt
$ file mojibake.txt
mojibake.txt: Non-ISO extended-ASCII text, with CRLF line terminators
$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> fp=open("mojibake.txt", "rb")
>>> original_string = fp.read()
>>> original_string
b' \x83t\x83@\x83C\x83\x8b\x82\xaa\x82P\x82\xc2\x82\xe0\x82\xa0\x82\xe8\x82\xdc\x82\xb9\x82\xf1\r\n'
>>> original_string.decode("shift-jis")
' ファイルが１つもありません\r\n'
>>>
I still can’t understand Japanese, but Google Translate® tells me this means “There are no files”.
Drilling down the listing part of AFSMAKE.EXE, it becomes apparent that this executable contains what is essentially a fully-featured version of afs.o as covered before, but compiled for Windows and using the C standard library I/O functions instead of the bespoke disc file layer.
It’s close enough that the differences are probably due to macros in the original code used as a compatibility layer between the PlayStation and Windows environments.
Therefore, my first mistake was supplying the suffix .VOL, as the AFS code concatenates it to the supplied path.
Let’s try again without the suffix:
$ wine AFSMAKE.EXE -l TENCHU/DATA
âtâ@âCâïé¬éPé┬éαéáéΦé▄é╣é±
No dice. What’s going on?
Further analysis reveals that the buffer used to hold said path is at most 16 bytes long and AFS rejects the user-supplied path if it’s more than 8 bytes long. That’s not for the filename of the archive, but for the whole path to the archive.
Even disregarding LFN (long filenames) which were introduced back in Windows 95, MS-DOS 2.0 supported directories back in 1983. Heck, the MIPS version of this code rejects paths longer than 75 bytes.
Fine, have it your way. Let’s try that again without that pesky directory in the way:
$ wine AFSMAKE.EXE -l DATA
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_J.TIM
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TIM.TPD
K:\WORK\CDIMAGE\STAGE\YAMIJOU\MAP.MAD
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_I.TIM
K:\WORK\CDIMAGE\STAGE\YAMIJOU\STAGE.CON
...
K:\WORK\CDIMAGE\TRIAL\THEME\MATO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\THEME9.TPD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\NOKI.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO2.TMD
1048 file(s)$
Finally, some conclusive results.
AFSMAKE.EXE doesn’t output a newline character on the file count line, so the shell prompt isn’t on a new line.
Now we’ve managed to list the contents of an archive with AFSMAKE.EXE, how about extracting some data?
$ wine AFSMAKE.EXE -x DATA 'K:\WORK\CDIMAGE\DEMO\START\CARD_E.TXT' card_e.txt
$ cat card_e.txt
Memory Card not found.
Memory Card damaged.
Memory Card not formatted.\Do you want to format it?
There is no saved data\for this game.
Memory Card full.
...
Couldn't read saved data.
Memory Card not found.\Game data can't be saved.\Okay?
Memory Card damaged.\Game data can't be saved.\Okay?
Memory Card doesn't have\enough space.\Game data can't be saved.\Okay?
Saving will overwrite\the previous save data.\Save anyway?
It works, but AFSMAKE.EXE can only extract one file at a time, which means that my abomination from part 6 is sadly not obsolete yet for extracting all the files from AFS archives.
For the last feature, we actually have a relevant artifact found during the previous part from the original developers, VOLMAKE.BAT.
It was most likely responsible for building the AFS archive as part of the original game’s development pipeline:
k:
cd \work\cdimage
del ..\data.vol
del data.vol
afsmake -p ..\data *
copy ..\data.vol .
… Oh, right, the game expects that everything inside the archive is under K:\WORK\CDIMAGE.
On Linux, we can define drive letters for Wine by creating symbolic links inside ~/.wine/dosdevices:
$ ln -s ~/Documents/K: ~/.wine/dosdevices/k:
$ cd ~/Documents/K:
$ wine cmd.exe
Microsoft Windows 6.1.7601
K:\>dir
Volume in drive K has no label.
Volume Serial Number is 0000-0000
Directory of K:\
3/10/2024 11:43 AM <DIR> WORK
0 files 0 bytes
1 directory 5,356,400,640 bytes free
K:\>
That exact trick won’t work on Windows since this relies on a feature specific to Wine.
It would probably require either having a K: drive or using a mount point to work around this issue.
With the K: drive available, we can try and create an AFS archive to run with the game:
$ WINEPATH=~/.local/bin wine cmd.exe
Microsoft Windows 6.1.7601
K:\>cd work\cdimage
K:\work\CDIMAGE>afsmake -p ..\DATA *
AFS Volume Maker Version 2.00
Create New volume... ..\DATA
add: K:\work\CDIMAGE\ANIM\ENGLISH\STAGE10A.CAD
add: K:\work\CDIMAGE\ANIM\ENGLISH\STAGE10R.CAD
add: K:\work\CDIMAGE\ANIM\ENGLISH\STAGE11A.CAD
add: K:\work\CDIMAGE\ANIM\ENGLISH\STAGE11R.CAD
add: K:\work\CDIMAGE\ANIM\ENGLISH\STAGE12A.CAD
...
add: K:\work\CDIMAGE\TRIAL\THEME\YATATE.TMD
add: K:\work\CDIMAGE\TRIAL\THEME\YKAGARI.TMD
add: K:\work\CDIMAGE\TRIAL\THEME\YOROI.TMD
add: K:\work\CDIMAGE\TRIAL\THEME\YUKIMORI.TMD
add: K:\work\CDIMAGE\TRIAL\THEME\ZIZOU.TMD
ok. update 1047 file(s).
K:\work\CDIMAGE>
That’s every feature of AFSMAKE.EXE exercised.
However, it’s not very useful unless we can get the game to use newly created archives.
While I could try and get the PCdrv backend of the game’s virtual file system working once more, it doesn’t use AFS archives and therefore wouldn’t show off AFSMAKE.EXE in action.
Therefore, let’s try repacking a modified version of the game.
We have the means to create new AFS archives, but we need to create a new ISO file for DuckStation.
PlayStation CD-ROM images are special enough that conventional tools such as xorriso won’t cut it, so I’ll use MKPSXISO, a purpose-built tool.
Sadly, the Linux version is built for Ubuntu and won’t work on this Debian distribution, so I’ll use the Windows version instead.
First, we need to generate a XML specification for the CD image to that the tool knows what to do.
We can just use dumpsxiso to rip the game and generate it for us:
$ WINEPATH=~/.local/bin wine cmd.exe
Microsoft Windows 6.1.7601
Z:\home\boricj\Documents\tenchu-hacked>dumpsxiso -s spec.xml "..\..\ISOs\Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan) (Track 1).bin"
DUMPSXISO 2.04 - PlayStation ISO dumping tool
2017 Meido-Tek Productions (John "Lameguy" Wilbert Villamor/Lameguy64)
2020 Phoenix (SadNES cITy)
2021-2022 Silent, Chromaryu, G4Vi, and spicyjpeg
ISO descriptor:
System ID : PLAYSTATION
Volume ID :
Volume Set ID :
Publisher ID :
Data Prep. ID :
Application ID : PLAYSTATION
Volume Create Date : 1999010214210800+36
Volume Modify Date : 0000000000000000+0
Volume Expire Date : 0000000000000000+0
ISO contents:
Extracting SYSTEM.CNF;1...
SYSTEM.CNF
Extracting SLPS_019.01;1...
SLPS_019.01
...
TENCHU\MOVIE\ENDAYA.STR
Extracting ENDING.STR;1...
TENCHU\MOVIE\ENDING.STR
Extracting CD.CA;1...
TENCHU\CD.CA
WARNING: The CDDA file TENCHU\CD.CA is out of the iso file bounds.
This usually means that the game has audio tracks, and they are on separate files.
As DUMPSXISO does not support dumping from a cue file, you should use an iso file containing all tracks.
DUMPSXISO will write the file as a dummy (silent) cdda file.
This is generally fine, when the real CDDA file is also a dummy file.
If it is not dummy, you WILL lose this audio data in the rebuilt iso.
Z:\home\boricj\Documents\tenchu-hacked>
It’s complaining about the second audio track easter egg that we found in part 2, but we don’t need it for our proof-of-concept so we’ll ignore that error message.
After replacing TENCHU\DATA.VOL with our modified version, we can repack the game with mkpsxiso:
Z:\home\boricj\Documents\tenchu-hacked>mkpsxiso -y -o "..\..\ISOs\Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan) (Hacked).iso" spec.xml
MKPSXISO 2.04 - PlayStation ISO Image Maker
2017-2022 Meido-Tek Productions (John "Lameguy" Wilbert Villamor/Lameguy64)
2021-2022 Silent, Chromaryu, G4Vi, and spicyjpeg
Building ISO Image: ..\..\ISOs\Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan) (Hacked).bin + mkpsxiso.cue
Scanning tracks...
Track #1 data:
Identifiers:
System : PLAYSTATION
Application : PLAYSTATION
Creation Date : 1999010214210800+36
License file: license_data.dat
Parsing directory tree...
Files Total: 24
Directories: 3
Total file system size: 739972128 bytes (314614 sectors)
Track #2 audio:
DA File CD.CA
Writing ISO
Writing files...
Packing SYSTEM.CNF... Done.
Packing SLPS_019.01... Done.
Packing TENCHU\RUN.EXE... Done.
Packing TENCHU\MENU.EXE... Done.
Packing TENCHU\MAIN.EXE... Done.
Packing TENCHU\TRIAL.EXE... Done.
Packing TENCHU\ENDING.EXE... Done.
Packing TENCHU\DATA.VOL... Done.
Packing XA TENCHU\MOVIE\SME.STR... Done.
Packing XA TENCHU\MOVIE\ACQUIRE.STR... Done.
Packing XA TENCHU\MOVIE\OPEN06.STR... Done.
Packing XA TENCHU\MOVIE\TRAINING.STR... Done.
Packing XA TENCHU\XA\TORA.XA... Done.
Packing XA TENCHU\XA\MUSIC.XA... Done.
Packing XA TENCHU\XA\STAGES.XA... Done.
Packing XA TENCHU\XA\EVENT_F.XA... Done.
Packing XA TENCHU\XA\EVENT_E.XA... Done.
Packing XA TENCHU\XA\EVENT_I.XA... Done.
Packing XA TENCHU\XA\EVENT_J.XA... Done.
Packing XA TENCHU\XA\INTRO.XA... Done.
Packing XA TENCHU\MOVIE\ENDRIKI.STR... Done.
Packing XA TENCHU\MOVIE\ENDAYA.STR... Done.
Packing XA TENCHU\MOVIE\ENDING.STR... Done.
Writing CDDA tracks...
Packing audio TENCHU/CD.WAV... Done.
Writing license data...Ok.
Writing directories... Ok.
ISO image generated successfully.
Total image size: 747785472 bytes (317936 sectors)
With this, we have an ISO file that’ll boot inside DuckStation.
We’ve repacked our modified version of Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen, it’s time to try it out. While inside a level, by pressing the Select button the player can display a map:
Each level has a map, whose file is named CHIZU.TIM.
For the first level Punish the Evil Merchant, this file is located inside the AFS archive at K:\WORK\CDIMAGE\STAGE\AKINDO\CHIZU.TIM.
The original Japanese release of Tenchu has different, hand-drawn maps than the newer releases.
In this mod, I’ve replaced that file with the version from Rittai Ninja Katsugeki Tenchu and we can see the result in-game:
The newer releases display a small red cross that follows the player on the map. The original release didn’t and its maps weren’t accurate to the level’s geometry, which is why the location of the small red cross is wrong relative to the hand-drawn map.
Hardly the mod of the century, but the proof-of-concept is done.
We’ve tested out AFSMAKE.EXE found inside the demo version of Rittai Ninja Katsugeki Tenchu, discovered it is quite janky, but nevertheless managed to use it to create a modified version of Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen.
Back in part 2, we’ve identified TENCHU/DATA.VOL as an archive file in a proprietary format.
We could go ahead and reverse-engineer that file format the traditional way, but that’s not how this series rolls.
Browsing the strings of MAIN.EXE for any interesting debugging messages, we can find a bunch of them prefixed with Afs and the first 8 bytes of TENCHU/DATA.VOL are AFS_VOL_200.
Coincidence? I think not.
The archive file of Tenchu: Stealth Assassins covers most of the game’s data (besides music, movies and executables), but the game doesn’t directly access it. Instead, it goes through a surprisingly sophisticated indirection mechanism:
There are three data sources implemented in the VFS code:
libsn functions such as PCopen() and PCread() that only work on a dev kit ;Interestingly, the AFS code doesn’t directly interface with Sony’s libcd library, but goes through an extra layer that provides access to files on the disc through file descriptors.
Retail versions are hard-coded to always initialize the VFS with AFS, since the other two would’ve been useful only during development and none of the other options will work on a standard PlayStation anyway. The memory data source is stubbed out on all versions but the Japanese demo, however the code for the PC data source is still present on all versions.
Currently we only have the AFS archive to work with and we don’t have any data in the correct format to feed the other data sources. We’ll need to extract the AFS archive first to get a hold of the assets.
After some reverse-engineering and annotating, we can identify the following address ranges of MAIN.EXE as part of the AFS archive code:
| Section | Address range |
|---|---|
.rdata |
0x80014870 - 0x800149f2 |
.text |
0x8005e950 - 0x8005f227 |
.sdata |
0x80098e78 - 0x80097e7f |
Just like the memory allocator, we’ll steal repurpose the game’s original code to run inside our own test programs by delinking it into afs.o, an object file:
$ readelf --wide --file-header afs.o
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: MIPS R3000
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 52 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 40 (bytes)
Number of section headers: 9
Section header string table index: 8
$ readelf --wide --section-headers --string-dump=.comment afs.o
There are 9 section headers, starting at offset 0x34:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .strtab STRTAB 00000000 00019c 0003de 00 0 0 1
[ 2] .symtab SYMTAB 00000000 00057c 0003c0 10 1 34 4
[ 3] .rdata PROGBITS 00000000 000940 000183 00 A 0 0 16
[ 4] .text PROGBITS 00000000 000ad0 000928 00 AX 0 0 16
[ 5] .sdata PROGBITS 00000000 001400 000008 00 WA 0 0 16
[ 6] .rel.text REL 00000000 001408 0002d0 08 I 2 4 4
[ 7] .comment PROGBITS 00000000 0016d8 000024 01 MS 0 0 1
[ 8] .shstrtab STRTAB 00000000 0016fc 000042 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)
String dump of section '.comment':
[ 0] ghidra-delinker-extension v0.3.0
$ readelf --wide --symbols afs.o
Symbol table '.symtab' contains 60 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 FILE LOCAL DEFAULT ABS afs.o
2: 00000000 0 SECTION LOCAL DEFAULT 3
3: 00000000 0 SECTION LOCAL DEFAULT 4
4: 00000000 0 SECTION LOCAL DEFAULT 5
5: 00000000 28 OBJECT LOCAL DEFAULT 3 s_AfsOpenVolume:_%s_open_err_80014870
6: 0000001c 28 OBJECT LOCAL DEFAULT 3 s_AfsOpenVolume:_Header_error_8001488c
7: 00000038 27 OBJECT LOCAL DEFAULT 3 s_AfsOpenVolume:_Entry_error_800148a8
8: 00000054 12 OBJECT LOCAL DEFAULT 3 s_AFS_VOL_200_800148c4
9: 00000064 25 OBJECT LOCAL DEFAULT 3 s_AfsGetEntry:_empty_index_800148d4
10: 00000080 31 OBJECT LOCAL DEFAULT 3 s_AfsGetEnty:_memory_not_enough!_800148f0
11: 000000a0 32 OBJECT LOCAL DEFAULT 3 s_AfsGetEntry:_memory_not_enough!_80014910
12: 000000c0 20 OBJECT LOCAL DEFAULT 3 s_Illigal_index_data_80014930
13: 000000d4 28 OBJECT LOCAL DEFAULT 3 s_AfsInit:_not_enough_memory!_80014944
14: 000000f0 23 OBJECT LOCAL DEFAULT 3 s_AfsOpen:_%s_not_found_80014960
15: 00000108 29 OBJECT LOCAL DEFAULT 3 s_AfsOpen:_no_more_handle_[%s]_80014978
16: 00000128 25 OBJECT LOCAL DEFAULT 3 s_AfsClose:_invalid_handle_80014998
17: 00000144 28 OBJECT LOCAL DEFAULT 3 s_AfsFileSize:_invalid_handle_800149b4
18: 00000160 24 OBJECT LOCAL DEFAULT 3 s_AfsRead:_invalid_handle_800149d0
19: 00000178 11 OBJECT LOCAL DEFAULT 3 s_@%d_%.195s_800149e8
20: 000000b8 4 OBJECT LOCAL DEFAULT 4 LAB_8005ea08
21: 0000020c 4 OBJECT LOCAL DEFAULT 4 LAB_8005eb5c
22: 00000348 4 OBJECT LOCAL DEFAULT 4 LAB_8005ec98
23: 00000388 4 OBJECT LOCAL DEFAULT 4 LAB_8005ecd8
24: 00000410 4 OBJECT LOCAL DEFAULT 4 LAB_8005ed60
25: 00000438 4 OBJECT LOCAL DEFAULT 4 LAB_8005ed88
26: 0000051c 4 OBJECT LOCAL DEFAULT 4 LAB_8005ee6c
27: 00000584 4 OBJECT LOCAL DEFAULT 4 LAB_8005eed4
28: 00000660 4 OBJECT LOCAL DEFAULT 4 LAB_8005efb0
29: 0000066c 4 OBJECT LOCAL DEFAULT 4 LAB_8005efbc
30: 000006a8 4 OBJECT LOCAL DEFAULT 4 LAB_8005eff8
31: 000006e8 4 OBJECT LOCAL DEFAULT 4 LAB_8005f038
32: 000007a0 4 OBJECT LOCAL DEFAULT 4 LAB_8005f0f0
33: 00000000 5 OBJECT LOCAL DEFAULT 5 s_.VOL_80097e78
34: 00000000 564 FUNC GLOBAL DEFAULT 4 AfsGetEntry
35: 00000234 560 FUNC GLOBAL DEFAULT 4 AfsFind
36: 00000464 204 FUNC GLOBAL DEFAULT 4 AfsOpenVolume
37: 00000530 100 FUNC GLOBAL DEFAULT 4 AfsInit
38: 00000594 80 FUNC GLOBAL DEFAULT 4 FUN_8005eee4
39: 000005e4 156 FUNC GLOBAL DEFAULT 4 AfsOpen
40: 00000680 56 FUNC GLOBAL DEFAULT 4 AfsClose
41: 000006b8 64 FUNC GLOBAL DEFAULT 4 AfsFileSize
42: 000006f8 196 FUNC GLOBAL DEFAULT 4 AfsRead
43: 000007bc 180 FUNC GLOBAL DEFAULT 4 FUN_8005f10c
44: 00000870 184 FUNC GLOBAL DEFAULT 4 AfsCheckMagic
45: 00000000 0 NOTYPE GLOBAL DEFAULT UND AdtMessageBox
46: 00000000 0 NOTYPE GLOBAL DEFAULT UND DiscOpenFile
47: 00000000 0 NOTYPE GLOBAL DEFAULT UND DiscReadFile
48: 00000000 0 NOTYPE GLOBAL DEFAULT UND DiscSeekFile
49: 00000000 0 NOTYPE GLOBAL DEFAULT UND MemoryAllocate
50: 00000000 0 NOTYPE GLOBAL DEFAULT UND MemoryFree
51: 00000000 0 NOTYPE GLOBAL DEFAULT UND memset
52: 00000000 0 NOTYPE GLOBAL DEFAULT UND sprintf
53: 00000000 0 NOTYPE GLOBAL DEFAULT UND strcat
54: 00000000 0 NOTYPE GLOBAL DEFAULT UND strcmp
55: 00000000 0 NOTYPE GLOBAL DEFAULT UND strcpy
56: 00000000 0 NOTYPE GLOBAL DEFAULT UND strlen
57: 00000000 0 NOTYPE GLOBAL DEFAULT UND strncmp
58: 00000000 0 NOTYPE GLOBAL DEFAULT UND strncpy
59: 00000000 0 NOTYPE GLOBAL DEFAULT UND toupper
$ readelf --wide --relocs afs.o
Relocation section '.rel.text' at offset 0x1408 contains 90 entries:
Offset Info Type Sym. Value Symbol's Name
00000038 00000905 R_MIPS_HI16 00000064 s_AfsGetEntry:_empty_index_800148d4
0000003c 00002d04 R_MIPS_26 00000000 AdtMessageBox
00000040 00000906 R_MIPS_LO16 00000064 s_AfsGetEntry:_empty_index_800148d4
00000044 00001504 R_MIPS_26 0000020c LAB_8005eb5c
00000050 00003104 R_MIPS_26 00000000 MemoryAllocate
00000064 00000a05 R_MIPS_HI16 00000080 s_AfsGetEnty:_memory_not_enough!_800148f0
00000068 00002d04 R_MIPS_26 00000000 AdtMessageBox
0000006c 00000a06 R_MIPS_LO16 00000080 s_AfsGetEnty:_memory_not_enough!_800148f0
00000070 00003204 R_MIPS_26 00000000 MemoryFree
00000078 00001504 R_MIPS_26 0000020c LAB_8005eb5c
00000090 00003104 R_MIPS_26 00000000 MemoryAllocate
000000a4 00000b05 R_MIPS_HI16 000000a0 s_AfsGetEntry:_memory_not_enough!_80014910
000000a8 00001404 R_MIPS_26 000000b8 LAB_8005ea08
000000ac 00000b06 R_MIPS_LO16 000000a0 s_AfsGetEntry:_memory_not_enough!_80014910
000000b0 00000c05 R_MIPS_HI16 000000c0 s_Illigal_index_data_80014930
000000b4 00000c06 R_MIPS_LO16 000000c0 s_Illigal_index_data_80014930
000000b8 00002d04 R_MIPS_26 00000000 AdtMessageBox
000000c0 00001504 R_MIPS_26 0000020c LAB_8005eb5c
000000d0 00003004 R_MIPS_26 00000000 DiscSeekFile
000000ec 00002f04 R_MIPS_26 00000000 DiscReadFile
000001ac 00003a04 R_MIPS_26 00000000 strncpy
00000200 00003204 R_MIPS_26 00000000 MemoryFree
00000268 00003a04 R_MIPS_26 00000000 strncpy
00000288 00003b04 R_MIPS_26 00000000 toupper
00000300 00003904 R_MIPS_26 00000000 strncmp
00000354 00003704 R_MIPS_26 00000000 strcpy
00000360 00001305 R_MIPS_HI16 00000178 s_@%d_%.195s_800149e8
00000364 00001306 R_MIPS_LO16 00000178 s_@%d_%.195s_800149e8
0000036c 00003404 R_MIPS_26 00000000 sprintf
00000374 00001704 R_MIPS_26 00000388 LAB_8005ecd8
0000037c 00001604 R_MIPS_26 00000348 LAB_8005ec98
000003c8 00003904 R_MIPS_26 00000000 strncmp
00000418 00001904 R_MIPS_26 00000438 LAB_8005ed88
00000420 00001804 R_MIPS_26 00000410 LAB_8005ed60
00000478 00002504 R_MIPS_26 00000530 AfsInit
00000480 00003804 R_MIPS_26 00000000 strlen
00000498 00003704 R_MIPS_26 00000000 strcpy
000004a4 00002105 R_MIPS_HI16 00000000 s_.VOL_80097e78
000004a8 00003504 R_MIPS_26 00000000 strcat
000004ac 00002106 R_MIPS_LO16 00000000 s_.VOL_80097e78
000004b4 00002e04 R_MIPS_26 00000000 DiscOpenFile
000004c4 00000505 R_MIPS_HI16 00000000 s_AfsOpenVolume:_%s_open_err_80014870
000004c8 00000506 R_MIPS_LO16 00000000 s_AfsOpenVolume:_%s_open_err_80014870
000004cc 00002d04 R_MIPS_26 00000000 AdtMessageBox
000004d4 00001a04 R_MIPS_26 0000051c LAB_8005ee6c
000004dc 00002c04 R_MIPS_26 00000870 AfsCheckMagic
000004e8 00000605 R_MIPS_HI16 0000001c s_AfsOpenVolume:_Header_error_8001488c
000004ec 00002d04 R_MIPS_26 00000000 AdtMessageBox
000004f0 00000606 R_MIPS_LO16 0000001c s_AfsOpenVolume:_Header_error_8001488c
000004f4 00001a04 R_MIPS_26 0000051c LAB_8005ee6c
000004fc 00002204 R_MIPS_26 00000000 AfsGetEntry
0000050c 00000705 R_MIPS_HI16 00000038 s_AfsOpenVolume:_Entry_error_800148a8
00000510 00002d04 R_MIPS_26 00000000 AdtMessageBox
00000514 00000706 R_MIPS_LO16 00000038 s_AfsOpenVolume:_Entry_error_800148a8
00000550 00003104 R_MIPS_26 00000000 MemoryAllocate
00000560 00000d05 R_MIPS_HI16 000000d4 s_AfsInit:_not_enough_memory!_80014944
00000564 00002d04 R_MIPS_26 00000000 AdtMessageBox
00000568 00000d06 R_MIPS_LO16 000000d4 s_AfsInit:_not_enough_memory!_80014944
0000056c 00001b04 R_MIPS_26 00000584 LAB_8005eed4
0000057c 00003304 R_MIPS_26 00000000 memset
000005b4 00003b04 R_MIPS_26 00000000 toupper
000005fc 00002304 R_MIPS_26 00000234 AfsFind
00000610 00000e05 R_MIPS_HI16 000000f0 s_AfsOpen:_%s_not_found_80014960
00000614 00001c04 R_MIPS_26 00000660 LAB_8005efb0
00000618 00000e06 R_MIPS_LO16 000000f0 s_AfsOpen:_%s_not_found_80014960
0000062c 00001d04 R_MIPS_26 0000066c LAB_8005efbc
00000658 00000f05 R_MIPS_HI16 00000108 s_AfsOpen:_no_more_handle_[%s]_80014978
0000065c 00000f06 R_MIPS_LO16 00000108 s_AfsOpen:_no_more_handle_[%s]_80014978
00000660 00002d04 R_MIPS_26 00000000 AdtMessageBox
00000690 00001e04 R_MIPS_26 000006a8 LAB_8005eff8
00000698 00001005 R_MIPS_HI16 00000128 s_AfsClose:_invalid_handle_80014998
0000069c 00002d04 R_MIPS_26 00000000 AdtMessageBox
000006a0 00001006 R_MIPS_LO16 00000128 s_AfsClose:_invalid_handle_80014998
000006d0 00001f04 R_MIPS_26 000006e8 LAB_8005f038
000006d8 00001105 R_MIPS_HI16 00000144 s_AfsFileSize:_invalid_handle_800149b4
000006dc 00002d04 R_MIPS_26 00000000 AdtMessageBox
000006e0 00001106 R_MIPS_LO16 00000144 s_AfsFileSize:_invalid_handle_800149b4
00000724 00001205 R_MIPS_HI16 00000160 s_AfsRead:_invalid_handle_800149d0
00000728 00002d04 R_MIPS_26 00000000 AdtMessageBox
0000072c 00001206 R_MIPS_LO16 00000160 s_AfsRead:_invalid_handle_800149d0
00000730 00002004 R_MIPS_26 000007a0 LAB_8005f0f0
0000074c 00003004 R_MIPS_26 00000000 DiscSeekFile
0000077c 00002004 R_MIPS_26 000007a0 LAB_8005f0f0
00000788 00002f04 R_MIPS_26 00000000 DiscReadFile
00000808 00003904 R_MIPS_26 00000000 strncmp
00000888 00003004 R_MIPS_26 00000000 DiscSeekFile
00000898 00002f04 R_MIPS_26 00000000 DiscReadFile
000008a4 00000805 R_MIPS_HI16 00000054 s_AFS_VOL_200_800148c4
000008a8 00003604 R_MIPS_26 00000000 strcmp
000008ac 00000806 R_MIPS_LO16 00000054 s_AFS_VOL_200_800148c4
This time, we have more undefined symbols besides the usual standard C library functions:
MemoryAllocate, MemoryFree) ;AdtMessageBox) ;DiscOpenFile, DiscReadFile, DiscSeekFile).We have a couple more functions to shim, but it’s still manageable. We also need the corresponding header file for the C compiler to parse:
#pragma once
#include <stdint.h>
typedef struct {
uint16_t signature;
uint16_t type;
uint32_t location;
uint32_t size;
char field1_0xc;
char field2_0xd;
char field3_0xe;
char field4_0xf;
char name[20];
} afs_entry_t;
typedef struct {
int32_t used;
int32_t cursor;
afs_entry_t* entry;
} afs_file_descriptor_t;
typedef struct {
disc_file_descriptor_t *discDescriptor;
char field1_0x4;
char field2_0x5;
char field3_0x6;
char field4_0x7;
int32_t entriesOffset;
afs_entry_t* entries;
int32_t numberEntries;
int32_t field8_0x14;
afs_file_descriptor_t* fileDescriptors;
} afs_volume_t;
int AfsGetEntry(afs_volume_t* handle);
afs_entry_t* AfsFind(afs_volume_t* handle, const char* path, unsigned short flags);
int AfsOpenVolume(afs_volume_t* handle, const char* path);
void AfsInit(afs_volume_t* handle);
afs_file_descriptor_t* AfsOpen(afs_volume_t* handle, const char* path);
int AfsClose(afs_file_descriptor_t* fd);
int AfsRead(afs_volume_t* volume, afs_file_descriptor_t* fd, void* buffer, int length);
int AfsCheckMagic(afs_volume_t* handle);
We do not have a definition for disc_file_descriptor_t inside afs.h, as it is part of the disc file subsystem.
We will shim it anyways during our investigations in this part.
We have our object file, so we might as well do something interesting with it.
But first, we need to provide alternative implementations for the missing symbols before we can start using afs.o:
typedef struct disc_file_descriptor_t disc_file_descriptor_t;
void AdtMessageBox(const char* fmt, ...) {
va_list args;
va_start(args, fmt);
vprintf(fmt, args);
va_end(args);
}
disc_file_descriptor_t* DiscOpenFile(const char* path) {
return (disc_file_descriptor_t*) fopen(path, "r");
}
int DiscReadFile(disc_file_descriptor_t* handle, void* buffer, int length) {
fread(buffer, length, 1, (FILE*) handle);
return 0;
}
int DiscSeekFile(disc_file_descriptor_t* handle, int offset, int whence) {
int origin;
switch (whence) {
case 0:
origin = SEEK_SET;
break;
case 1:
origin = SEEK_CUR;
break;
case 2:
origin = SEEK_END;
break;
default:
abort();
}
return fseek((FILE*) handle, offset, origin);
}
With that out of the way, it’s time for everyone’s favorite activity: code necromancy.
Let’s start with a single-shot file extractor as a first step, similar to what the game internally does but dumping out the contents to the standard output:
static afs_volume_t vol;
int main(int argc, char* argv[]) {
if (AfsOpenVolume(&vol, argv[1]) == -1) {
exit(EXIT_FAILURE);
}
afs_file_descriptor_t* fd = AfsOpen(&vol, argv[2]);
if (fd == 0) {
exit(EXIT_FAILURE);
}
int length = AfsFileSize(&vol, fd);
void* buffer = MemoryAllocate(length);
AfsRead(&vol, fd, buffer, length);
AfsClose(fd);
write(1, buffer, length);
MemoryFree(buffer);
return EXIT_SUCCESS;
}
Let’s build it:
$ mipsel-linux-gnu-gcc -g -static -fno-pic -no-pie -o test-afs.elf test-afs.c afs.o memory.o panic.o
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: afs.o: warning: linking abicalls files with non-abicalls files
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: memory.o: warning: linking abicalls files with non-abicalls files
/usr/lib/gcc-cross/mipsel-linux-gnu/10/../../../../mipsel-linux-gnu/bin/ld: panic.o: warning: linking abicalls files with non-abicalls files
As usual, the toolchain isn’t happy, but it does produce an executable.
Looking at MAIN.EXE, we can find the string K:\WORK\CDIMAGE\DEMO\start\card_j.txt embedded within it.
Let’s try with it:
$ qemu-mipsel ./test-afs.elf TENCHU/DATA 'K:\WORK\CDIMAGE\DEMO\start\card_j.txt'
[J[hĂ܂.
[J[hĂ܂.
tH[}bgĂ܂\tH[}bg܂?
̃Q[̃f[^\Z[uĂ܂.
[J[h\eʂ܂.
f[^ǂݍݒł\J[hȂʼn!
f[^ݒł\J[hȂʼn!
ǂݍ݊.
݊.
tH[}bgł\J[hȂʼn!
tH[}bg.
tH[}bgł܂ł.
f[^ǂݍ݂ł܂ł.
f[^݂ł܂ł.
ȑÕf[^Z[uĂ܂\㏑܂?
[J[h`FbNł!
tH[}bgĂ܂.
[J[hĂ܂\J[hĂȂ\f[^Z[uł܂.
[J[h\eʂ܂\̂܂܂ł̓Z[uł܂.
Z[uł܂낵ł?
Z[uf[^ǂݍݒł\J[hȂʼn!
Z[uf[^ݒł\J[hȂʼn!
Z[uf[^\ǂݍ݂ł܂ł.
[J[hĂ܂\Z[uł܂낵ł?
[J[hĂ܂\Z[uł܂낵ł?
[J[h\eʂ܂\Z[uł܂낵ł?
łɃf[^Z[uĂ܂\㏑\Õf[^͖Ȃ܂\Z[u܂?
VɃZ[u܂?
[J[hɃZ[u܂?
㏑܂?
f[^ǂݍ݂ł܂ł\x݂܂?
f[^݂ł܂ł\x݂܂?
̔Cf[^폜܂?
{ɂ낵ł?
폜ł\J[hȂʼn!
폜.
폜ł܂ł.
[J[hĂ܂\ҏWł܂낵ł?
[hł܂낵ł?
[J[h\tH[}bgĂ܂\ҏWł܂낵ł?
̃Q[̖{҃f[^\Z[uĂ܂.
̔Cf[^\Z[uĂ܂.
t@C܂ł.
ׂẴQ[f[^\iō쐬Cf[^ȊOj\㏑܂?
Well, that’s a whole lot of mojibake, but we can discern some structure with the dots and question marks.
I could try and find in what code page this is, but rather than wrestling with obsolete character encodings in a foreign language I’ll hazard a guess and try changing the file name to card_e.txt:
$ qemu-mipsel ./test-afs.elf TENCHU/DATA 'K:\WORK\CDIMAGE\DEMO\start\card_e.txt'
Memory Card not found.
Memory Card damaged.
Memory Card not formatted.\Do you want to format it?
There is no saved data\for this game.
Memory Card full.
Reading data.\Don't remove Memory Card!
Writing data.\Don't remove Memory Card!
Finished reading data.
Finished writing data.
Formatting.\Don't remove Memory Card!
Finished formatting.
Formatting failed.
Couldn't read data.
Couldn't write data.
Overwrite\previously saved data?
Checking Memory Card!
Memory Card not formatted.
Memory Card not found.\Insert a Memory Card\for saving data.
Memory Card does not have\enough space for saving data.
Game data can't be saved.\Okay?
Reading data.\Don't remove Memory Card!
Writing data.\Don't remove Memory Card!
Couldn't read saved data.
Memory Card not found.\Game data can't be saved.\Okay?
Memory Card damaged.\Game data can't be saved.\Okay?
Memory Card doesn't have\enough space.\Game data can't be saved.\Okay?
Saving will overwrite\the previous save data.\Save anyway?
Ah, a plain ASCII file that my terminal emulator can render and that I can read. I guess that the first file was also extracted correctly but my terminal couldn’t render whatever 90’s nonsense this was.
Unfortunately, none of the AFS functions from the game can enumerate the contents of the archive, so we can only extract files this way if we know their full path ahead of time. Therefore, we do need to take a peek at the underlying code and file format to figure this part out.
AFS archives contain a flat index table of files and directories. The first entry is a directory with a straightforward name, but the next ones encode the parent’s entry index in the name.
For example, the following directory tree layout:
Would be encoded like so in the index table:
| Index | Name | Real name | Parent index |
|---|---|---|---|
| 0 | K: |
K: |
N/A |
| 1 | @0_WORK |
WORK |
0 |
| 2 | @1_CDIMAGE |
CDIMAGE |
1 |
| 3 | @2_STAGE |
STAGE |
2 |
| 4 | @3_YAMIJO |
YAMIJO |
3 |
| 5 | @4_TITLE_J.TIM |
TITLE_J.TIM |
4 |
| 6 | @4_TIM.TPD |
TIM.TPD |
4 |
| … | … | … | … |
| 19 | @4_IE_3 |
IE_3 |
4 |
| 18 | @4_IE_2 |
IE_2 |
4 |
| 20 | @3_TEMPLE |
TEMPLE |
3 |
| 21 | @20_TITLE_J.TIM |
TITLE_J.TIM |
20 |
| 22 | @20_TITLE_I.TIM |
TITLE_I.TIM |
20 |
| … | … |
Needless to say, encoding the parent directory’s index number as part of the file name is super weird.
I haven’t studied the AFS code in depth, but the mere fact it’s calling sprintf as part of searching for an entry should’ve been a code smell.
Fortunately, we can hack our way around this mess by recomputing the depth of the entries and storing it in an unused field of afs_entry_t and overwriting the buffer for the path as we iterate through the index:
void walk_afs_volume(afs_volume_t* vol, void(*callback)(afs_volume_t* vol, char* name, int isfile)) {
int depth = 0;
char path[128] = { 0 };
strcpy(path, vol->entries[0].name);
strcat(path, PATH_SEPARATOR);
vol->entries[0].field1_0xc = 0;
for (int index = 1; index < vol->numberEntries; index++) {
afs_entry_t* entry = &vol->entries[index];
afs_entry_t* parent = &vol->entries[atoi(entry->name + 1)];
char* component = strchr(entry->name, '_') + 1;
char* target = path;
for (int count = parent->field1_0xc; count >= 0; count--) {
target = strstr(target, PATH_SEPARATOR) + strlen(PATH_SEPARATOR);
}
strcpy(target, component);
if (entry->location == 0xCDCDCDCD) {
entry->field1_0xc = parent->field1_0xc + 1;
callback(vol, path, 0);
strcat(path, PATH_SEPARATOR);
}
else {
callback(vol, path, 1);
}
}
}
At this point, writing a brand-new extractor from scratch would’ve probably been a better idea than insisting on piggy-backing the original game code.
At any rate, it makes for a decent shakedown of afs.o.
Regardless, with a means to enumerate complete paths we can call AfsOpen with those and extract them like in the previous section.
After writing some more code we end up with the unafs.elf utility, with command-line options inspired from tar.
It can list the files in a volume:
$ qemu-mipsel ./unafs.elf -tvf TENCHU/DATA
K:
K:\WORK
K:\WORK\CDIMAGE
K:\WORK\CDIMAGE\STAGE
K:\WORK\CDIMAGE\STAGE\YAMIJOU
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_J.TIM
K:\WORK\CDIMAGE\STAGE\YAMIJOU\TIM.TPD
...
K:\WORK\CDIMAGE\STAGE\CAVE\IE_1
K:\WORK\CDIMAGE\STAGE\CAVE\<BA><CB>߰ <81>` STAGE.
K:\WORK\CDIMAGE\STAGE\CAVE\TITLE_E.TIM
...
K:\WORK\CDIMAGE\TRIAL\THEME\MATO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\THEME9.TPD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\NOKI.TMD
K:\WORK\CDIMAGE\TRIAL\THEME\UFO2.TMD
It looks like there’s some data corruption inside the Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen AFS archive…
But more interestingly, it can also extract the whole archive:
$ qemu-mipsel ./unafs.elf -s 1 -xvf TENCHU/DATA
WORK
WORK\CDIMAGE
WORK\CDIMAGE\STAGE
WORK\CDIMAGE\STAGE\YAMIJOU
WORK\CDIMAGE\STAGE\YAMIJOU\TITLE_J.TIM
WORK\CDIMAGE\STAGE\YAMIJOU\TIM.TPD
...
WORK\CDIMAGE\TRIAL\THEME\MATO.TMD
WORK\CDIMAGE\TRIAL\THEME\THEME9.TPD
WORK\CDIMAGE\TRIAL\THEME\UFO.TMD
WORK\CDIMAGE\TRIAL\THEME\NOKI.TMD
WORK\CDIMAGE\TRIAL\THEME\UFO2.TMD
That has to be one of the most convoluted ways to write an asset extractor for a game, but it works.
Interestingly enough, there are files in the archive that are bigger than the available heap memory inside MAIN.EXE.
That means we can’t use memory.o delinked from MAIN.EXE and must shim MemoryAllocate and MemoryFree with the C standard memory heap allocator instead.
What’s even more strange is that there’s apparently only one call to AfsOpen (coming from the VFS), which loads files into a buffer allocated dynamically from the memory heap allocated.
Either these files aren’t actually used by the game or something else’s going on.
The files for this part can be found here: tenchu1.tar.gz
Going back to the VFS layer, now that we have extracted the assets from the AFS archive we could theoretically use the PC backend…
Patching MAIN.EXE to initialize the VFS layer with the PC backend (while still calling CdInit as the game still expects a CD) is fairly straightforward and DuckStation even supports PCdrv (open Settings > Advanced, check Enable PCDrv under Tweaks/Hacks and set up the path for the PCDrv root directory).
Unfortunately, Tenchu uses backslashes as path separators and relies on case-insensitive paths, owning to the fact it was probably developed on Windows systems. DuckStation doesn’t transform the path in any way and since I’m using Linux, a case-sensitive system with forward slashes as path separators, the game can’t find its files:
$ ls '/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:/WORK/CDIMAGE/IMAGE/IMAGES.ARC'
'/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:/WORK/CDIMAGE/IMAGE/IMAGES.ARC'
$ duckstation-qt
...
[ 2688.2700] E(HandleSyscall): PCopen: Failed to open '/home/boricj/Games/Rittai Ninja Katsugeki - Tenchu - Shinobi Gaisen (Japan)/K:\WORK\CDIMAGE\IMAGE\images.arc'
Even on Windows it won’t work as-is, since the game prefixes any paths with K:\.
It’s likely that the PC backend could be made to work again with additional efforts to fix all these issues, but I can’t be bothered to do it right now.
If we use unafs on the demo version of Rittai Ninja Katsugeki Tenchu, things get really interesting.
Backups of files that were probably being modified at the time the demo was finalized, lots of files that the demo probably didn’t use and things that definitively shouldn’t be there:
K:\WORK\CDIMAGE\$RES_UP.BATK:\WORK\CDIMAGE\AFSMAKE.EXEK:\WORK\CDIMAGE\CD.CCSK:\WORK\CDIMAGE\CD.CTIK:\WORK\CDIMAGE\ENDING.EXEK:\WORK\CDIMAGE\GAME.EXEK:\WORK\CDIMAGE\LICENSEJ.DATK:\WORK\CDIMAGE\PSX.EXEK:\WORK\CDIMAGE\PSX.SYMK:\WORK\CDIMAGE\RESTART.EXEK:\WORK\CDIMAGE\VOLMAKE.BATThese files (except for LICENSEJ.DAT) can be found here: bonus.tar.gz
We have batch files, CD-ROM generation logs, a bunch of PlayStation executables, debugging symbols for PSX.EXE, the Sony PlayStation license file for Japan (that can be found inside the Psy-Q SDK) and finally AFSMAKE.EXE, a PE i386 console executable for Windows.
The mastering of the gold disc for the demo back left some unexpected surprises.
Turns out the Tenchu modding community already knew about these (and also had a working AFS extractor that isn’t scavenged from the corpse of an executable), but it appears this wasn’t publicly documented online before. Still, it’s good I’ve stumbled upon these things before I got any deeper inside my reverse-engineering effort.
We have twisted the original archive code of Rittai Ninja Katsugeki Tenchu: Shinobi Gaisen into a working asset extractor for the AFS file format. We’ve also found some noteworthy artifacts inside the demo version of Rittai Ninja Katsugeki Tenchu that warrant further examination.
]]>