International Patching System (IPS) is the de-facto standard for ROM patches. Many clone/FPGA systems support .ips files in one way or another. I assume anyone who ran into this page is at least familiar with them. I suspect the most common use is to play fan translations of games not localized in your region. That's about all I use them for and of course everyone else thinks exactly like me.
OK, I also have written a .ips creator before. It was in .NET for some ROM editing tools in the same language. Now I'd like to write code to read .ips files and patch ROMs using .ips files too. Along the way I'll write a test program to verify everything works as expected.
The whole testing process will go like:
Let's first review the IPS file format:
Bytes 0-4 = PATCH
Bytes 5-7 = Address of first patch record
Bytes 8-9 = Length of first path record
Bytes A-[length] = the data to patch
... Repeat address, length, data for however many records there are ...
Last 3 bytes = EOF
Very simple. If I was a computer science professor writing an implementation of this format would be the first assignment for the data structures unit.
Since I'm not a computer science professor, that's not why I started this project. I only wanted to play Lord Monarch in English:
Outside of the title screen this game is entirely in Japanese. There is a very good fan translation though. I can of course simply use an existing patcher, and will for test purposes, but what fun is that?
Lord Monarch is one of the handful of Falcom games I've never played. I don't know if I'll enjoy it but it's worth trying. The title screen says this was released for Mega Drive in 1994. That gave it a 0% chance of seeing a North America port. By late 1994 Sega's marketing was all-in on the 32X. By 1995 it was Saturn or bust. An extremely optimistic translation project might have seen this launch after the Saturn, effectively dooming it. Instead we had to wait 26 years for dedicated fans to localize it.
That's enough background, time to start coding...
The latest version of this code will be here. I tested all this code. I'll never say any code is bug-free but this all works as expected. It's possible that by the time you read this I will have added some features or rearranged stuff for no obvious reason. Those are both things I am prone to do. So if you want this code, get the version in Github.
We'll start by creating some constants. The first two are the header and footer values. The other three are error messages for later use.
public abstract class IPSConstants{
public final static byte[] HEADER={'P','A','T','C','H'};
public final static byte[] EOF={'E','O','F'};
public final static String ERROR_INVALID_HEADER="Missing required header value";
public final static String ERROR_INVALID_EOF="Missing required end of file terminator";
public final static String ERROR_INVALID_FILE="Not a valid IPS file";
}
Then it's a very simple class to represent an IPS record. It's just two fields since the data length doesn't need a dedicated field. The NumberFormatters import is there to make toString() look readable. Feel free to omit that if you use this code.
import com.huguesjohnson.dubbel.util.NumberFormatters;
public class IPSRecord{
private int offset;
private byte[] data;
public IPSRecord(int offset){
this.offset=offset;
}
public IPSRecord(int offset,byte[] data){
this.offset=offset;
this.data=data;
}
public int length(){
if(this.data==null){return(0);}
return(this.data.length);
}
@Override
public String toString(){
String newLine=System.lineSeparator();
StringBuilder sb=new StringBuilder();
sb.append("offset=");
sb.append(NumberFormatters.intToHex(this.offset));
sb.append(newLine);
sb.append("length()=");
sb.append(NumberFormatters.intToHex(this.length()));
sb.append(newLine);
sb.append(NumberFormatters.byteArrayToString(this.data));
return(sb.toString());
}
/* autogenerated code below */
public int getOffset() {
return offset;
}
public void setOffset(int offset) {
this.offset = offset;
}
public byte[] getData() {
return data;
}
public void setData(byte[] data) {
this.data = data;
}
}
Next is a class to write a file containing a List of IPS records. I'll probably move that intToBytes method to another utility class someday. Whether to handle, re-throw, or do nothing with Exceptions is always a spirited debate. In this case I opted to let the caller of writeIPSFile deal with anything that goes wrong.
import java.io.FileOutputStream;
import java.util.List;
public abstract class IPSWriter{
public final static void writeIPSFile(String filePath,List<IPSRecord> records) throws Exception{
FileOutputStream fout=null;
try{
fout=new FileOutputStream(filePath);
fout.write(IPSConstants.HEADER);
for(IPSRecord record:records){
byte[] offset=intToBytes(record.getOffset(),3);
fout.write(offset);
byte[] length=intToBytes(record.length(),2);
fout.write(length);
fout.write(record.getData());
}
fout.write(IPSConstants.EOF);
}
finally{
if(fout!=null){fout.flush();fout.close();}
}
}
public final static byte[] intToBytes(int i,int numBytes){
byte[] b=new byte[numBytes];
for(int j=0;j<numBytes;j++){
int shift=8*j;
b[numBytes-1-j]=(byte)((i>>shift)&0xFF);
}
return(b);
}
}
Still very simple so far, intToBytes was the most complicated logic.
The last time I worked with .ips files I created a writer but not a reader because I didn't need it for that project. This is also a trivial task, about half the code is performing validations.
import java.util.ArrayList;
import java.util.List;
import com.huguesjohnson.dubbel.file.FileUtils;
public abstract class IPSReader{
public static List<IPSRecord> readFile(String filePath) throws Exception{
ArrayList<IPSRecord> records=new ArrayList<IPSRecord>();
byte[] b=FileUtils.readBytes(filePath);
int length=b.length;
/*
* Min size for IPS file with single record
* Header=5
* Address=3
* Length=2
* Data=1
* EOF=3
*/
if(b.length<13){
throw(new Exception(IPSConstants.ERROR_INVALID_FILE));
}
if(
(b[0]!=IPSConstants.HEADER[0])||
(b[1]!=IPSConstants.HEADER[1])||
(b[2]!=IPSConstants.HEADER[2])||
(b[3]!=IPSConstants.HEADER[3])||
(b[4]!=IPSConstants.HEADER[4])
){
throw(new Exception(IPSConstants.ERROR_INVALID_HEADER));
}
int i=5;
while(i<length){
//offset is three bytes
int offset=0;
offset+=(Byte.toUnsignedInt(b[i])<<16);
i++;
offset+=(Byte.toUnsignedInt(b[i])<<8);
i++;
offset+=Byte.toUnsignedInt(b[i]);
i++;
//record length is two bytes
int recordLength=0;
recordLength+=(Byte.toUnsignedInt(b[i])<<8);
i++;
recordLength+=Byte.toUnsignedInt(b[i]);
i++;
byte[] data=new byte[recordLength];
for(int j=0;j<recordLength;j++){
data[j]=b[i];
i++;
}
records.add(new IPSRecord(offset,data));
//check if EOF
if(i+2>length){
throw(new Exception(IPSConstants.ERROR_INVALID_EOF));
}
if(
(b[i]==IPSConstants.EOF[0])&&
(b[i+1]==IPSConstants.EOF[1])&&
(b[i+2]==IPSConstants.EOF[2])
){i=length;}
}
return(records);
}
}
And now we need some patching code. I couldn't decide if I wanted to this code to (a) copy the ROM being patched or (b) patch the ROM in place. So I wrote both since one is sort of a wrapper for the other.
import java.io.File;
import java.io.RandomAccessFile;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.util.List;
public abstract class IPSPatcher{
public static void copyAndPatch(String sourceFilePath,String destinationFilePath,String ipsPath) throws Exception{
Files.copy((new File(sourceFilePath)).toPath(),(new File(destinationFilePath)).toPath(),StandardCopyOption.REPLACE_EXISTING);
patchInPlace(destinationFilePath,ipsPath);
}
public static void patchInPlace(String fileToPatchPath,String ipsPath) throws Exception{
List<IPSRecord> records=IPSReader.readFile(ipsPath);
RandomAccessFile fout=null;
try{
fout=new RandomAccessFile(fileToPatchPath,"rw");
for(IPSRecord record:records){
fout.seek(record.getOffset());
fout.write(record.getData());
}
}
finally{
if(fout!=null){fout.close();}
}
}
}
Finally, some code to run through steps 3-9 listed at the start of this page.
Note - this code is not in Github.
import java.io.File;
import java.io.FileWriter;
import java.util.ArrayList;
import java.util.List;
import com.huguesjohnson.dubbel.file.FileUtils;
import com.huguesjohnson.dubbel.ips.IPSPatcher;
import com.huguesjohnson.dubbel.ips.IPSReader;
import com.huguesjohnson.dubbel.ips.IPSRecord;
import com.huguesjohnson.dubbel.ips.IPSWriter;
import com.huguesjohnson.dubbel.util.ByteComparer;
import com.huguesjohnson.dubbel.util.ByteRangeComparerResult;
public class IPSTest{
public static void main(String[] args){
//this is the path to the original unpatched ROM
String originalRomPath="Lord Monarch - Tokoton Sentou Densetsu (Japan).md";
//this is the path to the ROM patched using a known good patcher
String goodPatchRomPath="Lord Monarch - Tokoton Sentou Densetsu (Japan) (patched RH).md";
//this is out of laziness
int romLength=2097152;
//this is where the comparison between the previous two ROMs is written for debugging
String romDiffPath="romdiff.txt";
//this is the path to the IPS file created by all this Java code
String createIPSPath="Lord Monarch - Java Test.ips";
//this is the path to the original IPS file
String compareIPSPath="Lord Monarch - Tokoton Sentou Densetsu (Release 1.0).ips";
//this is the path to the ROM patched using this new Java code
String testPatchRomPath="Lord Monarch - Tokoton Sentou Densetsu (Japan) (patched Java).ips";
try{
//compare the two files and create an IPS file based on the differences
List<ByteRangeComparerResult> arrbrcr=ByteComparer.compareRange(originalRomPath,0,goodPatchRomPath,0,romLength);
ArrayList<IPSRecord> records=new ArrayList<IPSRecord>();
FileWriter writer=new FileWriter(new File(romDiffPath));
for(ByteRangeComparerResult brcr:arrbrcr){
writer.write(brcr.toString());
writer.write("\n");
IPSRecord ipsr=new IPSRecord(brcr.getFile2Line(),brcr.getFile2Value());
writer.write(ipsr.toString());
records.add(ipsr);
writer.write("\n");
}
writer.flush();
writer.close();
//write the ips file
IPSWriter.writeIPSFile(createIPSPath,records);
//now compare the IPS file created to the original
boolean same=FileUtils.compareFiles(new File(createIPSPath),new File(compareIPSPath));
if(same){
System.out.println("IPS files are the same, hooray!");
}else{
System.out.println("IPS files are not the same but it turns out that's ok.");
}
//read the file created and verify the records match
ArrayList<IPSRecord> verifyRecords=new ArrayList<IPSRecord>();
verifyRecords.addAll(IPSReader.readFile(createIPSPath));
same=true;
int size=records.size();
if(size!=verifyRecords.size()){
System.out.println("size="+size);
System.out.println("verifyRecords.size()="+verifyRecords.size());
same=false;
}
int i=0;
while(same&&(i<size)){
IPSRecord ips1=records.get(i);
IPSRecord ips2=verifyRecords.get(i);
if(ips1.getOffset()!=ips2.getOffset()){
System.out.println("ips1.getOffset()="+ips1.getOffset());
System.out.println("ips2.getOffset()="+ips2.getOffset());
same=false;
}else if(ips1.length()!=ips2.length()){
System.out.println("ips1.length()="+ips1.length());
System.out.println("ips2.length()="+ips2.length());
same=false;
}else{
byte[] b1=ips1.getData();
byte[] b2=ips2.getData();
int l=b1.length;
int j=0;
while(same&&(j<l)){
if(b1[j]!=b2[j]){
same=false;
System.out.println("b1[j]="+b1[j]);
System.out.println("b2[j]="+b2[j]);
}
j++;
}
}
i++;
}
if(same){
System.out.println("Validation of IPS file succeeded, hooray!");
}else{
System.out.println("Validation of IPS file failed.");
}
//test if the IPS file created creates the expected patched file
IPSPatcher.copyAndPatch(originalRomPath,testPatchRomPath,createIPSPath);
same=FileUtils.compareFiles(new File(goodPatchRomPath),new File(testPatchRomPath));
if(same){
System.out.println("Patched file is correct, hooray!");
}else{
System.out.println("Patched file is wrong, bad times.");
}
}catch(Exception x){
x.printStackTrace();
}
}
}
Perhaps you noticed the text that says "IPS files are not the same but it turns out that's ok". That was an interesting thing to debug. I created a .ips file based on deltas between the original ROM and a known good patch. Yet that file didn't match the .ips file used to create the patched ROM.
There's a very simple explanation for that:
See, isn't that obvious?
No, not really?
The author of the original .ips file has records that patch over bytes that are the same. Frequently where there is only a small gap between two offsets. I get it, I might have done the same. So they have a record that is 6 bytes long with the 3rd byte not changing. My code instead produced two records that are 2 and 3 bytes long respectively. Same result as proven by comparing ROMs patched with each .ips file.
Here are screenshots from the patch created with this code:
Although I could have faked these, you'll just have to take my word for it.
OK, this was fun. Maybe later I'll write a UI of some kind.
Related
