I've written things that are very niche. Some articles have probably made readers wonder if I'm not right. This article is going to be extraordinarily esoteric and disorganized, therefore reinforcing both of these. I'm fine BTW, quite possibly never been better. I've had a kind of an unproductive streak recently but finishing the code in this article just might turn it all around. That's not an exaggeration no matter how much it sounds like one.
The audience for this piece though, it doesn't exist. At best it's extremely small. That's never stopped me before so here we go...
This article will cover exciting topics like:
I hope that teaser worked.
The latest version of the code for this article is here. I only have a few snippets listed on this page.
A long time ago, later on I'll confirm it was shortly before August 2015, I decided to try writing a Sega Genesis demo. It took a few years to finish. It wasn't very good but it was a complete game that worked. That's something.
This was all fun in a warped sort of way so I made a couple other demos. Along the way I built something resembling a game engine and tools to generate code. There are too many repetitive tasks in developing any software. I found that building a game is especially repetitive. Things like palettes and tiles for example, they are very boring to create. I addressed this by writing code to generate these from image files. All the boring, repetitive things became image or json files that could be converted into 68k assembly code.
This is what the build tooling looks like as of the last demo I finished:
These are creatively named "build tools for retail clerk", because I couldn't think of anything better.
That box in the bottom corner is a giant hassle. It's slow and requires an oddball file sharing setup. I need to eliminate that.
This is what I set out to create:
The best solution would be running xm2esf locally. The tool is a mix of C++ and FreeBASIC code, the latter being supported on Windows only (hence this goofball virtualized solution). The second best solution might have been running it in WINE or something similar. I went with the third best solution of porting the code to Java.
If you have an application that works, and is free, then rewriting it is ill-advised (by me). My motivation is that the rest of my build tools are in Java and have been tested on multiple platforms. Even if I coerced xm2esf into running on a non-Windows platform that setup would be different on each operating system. I can't even guarantee it would be the same across Linux distributions. For whatever reason that's important to me. I also have some absurd ideas like making these build tools create Atari Jaguar games from the same source files. This would be a step in that direction. I first have to convince myself to spend $200 on an Atari Jaguar flash cart though.
There is of course another option I could have considered - writing my own Sega Genesis sound driver and tools to convert [some audio format] to work with it. This is what an actual smart person would do.
The file conversion part of xm2esf was written in FreeBASIC. The xm loader was written in C++ but I already wrote an xm loader in Java. So I only had to worry about the FreeBASIC code.
Like a whole lot of programmers my age, BASIC was the first language I learned. I worked as a Visual Basic programmer for a while and have a handful of free Visual Basic applications on this very site. I could have rewritten it by hand. Instead I decided to test out how well Google Gemini could handle this conversion.
There were a couple steps involved:
The resulting code was still structured like a BASIC program. It had one giant main method, dozens of global variables, and methods that manipulated global variables instead of using parameters. For example, let's start with a macro from the original code:
#MACRO WriteVol(a)
IF ctype(i) = 0 THEN
PUT #20, , chr(esfchan(i) + &h20)
TEMP = INT(fmvol(quotient(i) * a))
PUT #20, , chr(TEMP)
ELSE
IF ctype(i) = 3 THEN PUT #20, , chr(&H2A) ELSE PUT #20, , chr(esfchan(i) + &H20)
TEMP = INT(psgvol(quotient(i) * a))
PUT #20, , chr(TEMP)
END IF
#ENDMACRO
Macros aren't a thing in Java so it was converted to a method:
private static void writeVol(double a) throws IOException { // 'a' is volume value
int temp;
if (ctype[i] == 0) { // FM channel
esfOs.write((byte) (esfchan[i] + 0x20));
temp = (int) fmvol(quotient[i] * a);
esfOs.write((byte) temp);
} else { // PSG/Noise channel
if (ctype[i] == 3) { // Noise channel
esfOs.write((byte) 0x2A); // &H2A
} else {
esfOs.write((byte) (esfchan[i] + 0x20));
}
temp = (int) psgvol(quotient[i] * a);
esfOs.write((byte) temp);
}
}
See - one parameter, 'a' from the original, and everything else is a global variable. fmvol() and psgvol() are two actually independent functions. There are also some magic numbers. The most challenging part was the global variable simply called 'i'. In the original code it acts as a global loop control. The was the largest part of the logic I had to unwrap. I'm not knocking the original code here. It is fit for purpose code, AKA the best kind of code. It does exactly what it is supposed to do well.
Anyway, all these methods I refactored to look more like:
//FM and PSG logic are similar enough they could be grouped together instead of the current implementation
public static void writeVolume(OutputStream out,int esfChannel,ChannelType channelType,int volume,double quotient) throws IOException{
if(channelType==ChannelType.FM){
out.write((byte)(esfChannel+ESFEvent.SET_VOL_FM1.getValue()));
int fmVolume=(int)XmToEsfUtil.calculateFmVolume(quotient*(double)volume);
out.write((byte)fmVolume);
}else{
if(channelType==ChannelType.NOISE){
out.write(ESFEvent.SET_VOL_PSG3.getValue());
}else{//PSG
out.write((byte)(esfChannel+ESFEvent.SET_VOL_FM1.getValue()));
}
int psgVolume=(int)XmToEsfUtil.calculatePSGVolume(quotient*(double)volume);
out.write((byte)psgVolume);
}
}
It's functionally the same but more Java-y. Whether it's "better" than the original is completely subjective.
The original code relied on a custom mapping format and contained code to parse it. I converted that to JSON so it could be created & read through serialization (the general definition of serialization, not the specific Java meaning of serialization). The original code had some settings in the command-line which converted to:
// COMMAND$(3) to COMMAND$(255) in BASIC. We will parse 'args' from index 2 onwards
for (int cmdl = 2; cmdl < args.length; cmdl++) {
String arg = args[cmdl].toLowerCase(Locale.ROOT);
switch (arg) {
case "-r":
psgRetrigAll = 1;
break;
case "-p":
psgRetrig = 1;
break;
case "-i":
psgIgnore = 1;
break;
default:
// Unknown switch, might want to print warning
break;
}
}
I cut all that and moved these three things to the JSON mapping file.
In BASIC, arrays are indexed starting at 1. While Java arrays are indexed starting at 0. Gemini settled on making them one element larger and ignoring the 0th element. I re-wrote all of that. Maybe Gemini could have done that if I asked.
In some cases, Gemini added comments that helped explain the choices it made during conversion:
public static double calculateVibratoSlideStepConversion(double vibratoStep,int vibratoDepth,int currentNote){
/*
* Source code conversion notes from Gemini:
* C++: SIN(pi/180 * vibstep(i)) -- This implies 'pi' was actually used for trig in C++!
* The BASIC used 'fbmath.bi' which is FreeBASIC's math library.
* 'SIN' usually expects radians. 'pi/180 * angle_in_degrees' converts to radians.
* Or, if 'pi' from 'fbmath.bi' is defined as it would be used in trig, then 'Math.PI' is fine.
* Given 'pi/180', it's likely 'vibstep' is treated as degrees.
*/
double conversion=Math.sin(Math.toRadians(vibratoStep))*vibratoDepth/5.0+currentNote;
return(conversion);
}
There were some compile errors the first time. Gemini referenced variables that it didn't declare for example. That is easy enough to fix. Now it's time to test it all.
My test approach was simple - take a file produced by xm2esf and compare it to a file created through this converted code. I started with the title track to Retail Clerk '89 simply because it was first. Doing a byte comparison is easy, I already have the code to do that. That is not helpful for debugging this particular thing. I needed something that would help track down where a conversion went astray.
The Echo Stream Format is a list of events as defined here. What I needed was something to do an event-by-event comparison. That started by creating a whole lot of enums to represent all the supported events. After that, a little code to do the comparison.
None of this is throwaway code. The enums are needed to write ESF files... or to write ESF files with code that I might be able to understand next time I look at it. The comparison code is needed for debugging and to test that I didn't break anything when I next make changes.
To my surprise, the first file converted with the Java code matched the original. I thought that was it, all done, let us never speak of this again.
I went on comparing every previous xm2esf conversion. The first one worked, then the second, then the third, and even the fourth. It wasn't until the fifth file I compared that I ran into an issue:
Original -- Created by the converted code
[2884][Set volume FM channel #1][6][-1] -- [2884][Set volume FM channel #1][127][-1]
So at byte 2884 in one conversion something broke. In the original file it set the volume on FM channel 1 to 6 while the other muted it. I tracked it down to a specific line in the original XM pattern. Here it is:
What's going on here is a volume slide. The volume on channel 1 is fading down to zero. In the original conversion the event at byte 2884 is the volume turning back up (the bottom shaded region). In the code I just converted to Java it's still fading.
I next compared the raw ESF files and could see that the original stopped fading out once the volume went to zero and then skipped over the block of silence.
Those blocks of 217F227 are the volume slide and FE01FE01 is the silence. The 00AF220A is where the two files sync-up again.
Here's the actual audio output confirming this:
I of course assumed this was a problem with my conversion code and set off to debug it... over two weeks.
When I say I spent two weeks debugging a problem, it doesn't mean I spent 80+ hours debugging a problem. I have a regular job where I debug other problems. I have a family and occasional social activities. I also waste a lot of time at night playing video games I've already played. I add this disclaimer because there is nothing people on the internet enjoy more than "correcting" others. Nothing triggers this more than personal anecdotes on a completely free site. You'll have to trust me on this.
Anyway, so I spent about 10 hours over two weeks trying to figure out what was going on. Half of it was pinpointing where exactly the two ESF files fell out of sync. I summarized that to a 5 second snippet above. The other half was finding where exactly in the code this mistake could be happening. It took way too long for me to think of the most incredibly obvious thing...
Was the FreeBASIC code I converted the same code used to produce the original ESF files?
Spoiler alert: It was not.
I was inspired to try the idea of developing a demo on a classic console after attending a panel by David Crane at 2014 Classic Gaming Expo. He talked about the process of developing games like Pitfall! and A Boy and His Blob. I came away from that panel thinking it didn't sound all that difficult. It's a lot of trial & error and grinding. In other words, it's programming. That's everything I ever worked on. I knew I wasn't creative enough to make something as cool as the two aforementioned games, but I knew I could handle the work. It would be in my free time, it would take me years to finish something, it was also doable.
I didn't know which console I wanted to try developing for. I poked around with a few and by early 2015 I landed on the Sega Genesis. Around then I downloaded xm2esf and wrote an article about working with it. The audio conversions it produced sounded fine. I didn't notice that on some tracks the volume slide effects didn't match the original. I didn't even know what volume slide was. Just a few short months later, in August 2015, the developer of xm2esf made a large update. One thing it fixed was, you already know, issues with how volume slide effects were handled.
So my converted code was doing the right thing. I was comparing it against files generated by code doing the wrong thing. Luckily for my remaining sanity, I tested the conversion against 4 files that didn't use volume slide effects (or perhaps just fine volume slides specifically). If I ran into this issue right away I might have quit. Instead I knew it worked for 4 cases and just needed to debug this one thing.
Other than not thinking of this very simple solution, I made one other rookie mistake. After downloading xm2esf.exe I never looked to see if the developer made a newer version. I had something that worked and didn't want to risk it maybe? When I went to try this conversion I saw the original was last updated 7 years prior and thought that must be what I had without confirming.
This was all very dumb of me and I deserve to suffer 10 hours of misery for it. The converted code was working correctly the whole time.
I suppose I can get back to writing another Sega Genesis demo. I have two ideas, one seems relatively short while the other is probably too ambitious. It's also possible I'll update Retail Clerk '90 (and my other two demos) to use this updated tooling with some audio file fixes.
I also wonder now... could I run my old VB6 programs through Gemini and have it convert them to something zany like an Electron app?
Whatever I choose next, I'm just happy to be in place where I can start working on something new again.
Related
