I began writing GPU drivers in 2006 and had to support some legacy fixed-function chips from the late 90s at one point.

I think you and other commenters pretty much summarized what it was like. Documentation was often poor, so you would sometimes have to reach out to the folks who had actually written the hardware (or the simulator) for guidance.

That is the easy part of writing a driver, even today. Just follow the specification. The code in a GPU driver is relatively simple, and doesn't vary that much from one generation to the next. In the 90s some features didn't have hardware support, so the driver would do a bunch of math in the CPU instead, which was slow.

I'm contrast, the fun part are the times when the hardware deviates from he specification, or where the specification left things out and different people filled in the blanks with their own ideas. This is less common nowadays, as the design process has become more refined.

But yeah, debugging hardware bugs essentially boils down to:

(1) writing the simplest test that triggers the unexpected behavior that you had observed in a more complex application, then

(2) providing traces of it to the folks who wrote that part of the hardware or simulator,

(3) wait a few days for them to painstakingly figure out what is going wrong, clock by clock, and

(4) implement the workaround that they suggest, often something like "when X condition happens on chips {1.23, 1.24 and 1.25}, then program Y register to Z value, or insert a command to wait for module to complete before sending new commands".

It was more tedious than anything. Coming up with the simplest way to trigger the behavior could take weeks.

Well, that's what it was like to write user mode drivers. The kernel side was rather different and I wasn't directly exposed to it. Kernel drivers are conceptually simpler and significantly smaller in terms of lines of code, but much harder to debug.

You can download the Voodoo 2 programming manual which is only around 250-something pages long. The Voodoo2 was fixed function; you loaded assets into the voodoo memory then called functions to operate on those assets. The driver takes care of those two roles by loading and managing the assets in the voodoo memory and an API to program the registers on the card to execute functions on the loaded assets. There were more steps involved with geometry processing which happened in the driver but I am unsure if those were handled in user space by the libraries the application called or the driver code itself.

This isn't 250 something pages, only 132 so maybe I was wrong, but its a good look into how the Voodoo2 worked: https://www.dosdays.co.uk/media/3dfx/voodoo2.pdf

See also: https://3dfxarchive.com/reference.htm

A fun tidbit is the voodoo2 has a 2D mode but was not VESA compliant so it could not be used in a PC without being tied to a VESA card for 2D graphics. I believe that ability was there for custom and non-pc platforms.

Thanks, that’s interesting. I never owned a Voodoo card but I was definitely drooling over it when I saw a demo machine running Unreal on it.

And the second link definitely has everything one needs to do everything with those cards.

Thanks a lot! I'll take a look. It's really a treasure trove. But I probably need to purchase some Voodoo card to do real work on it.

Graphics programming "in the 90s" encompasses a pretty wide range.

You had raw access to things like VGA cards and then different tricks per each individual card when it came to things like SVGA

http://qzx.com/pc-gpe/index.php

You had a few things like UniVBE that tried to create a common platform for VESA cards to be used

https://wiki.sierrahelp.com/index.php/Scitech_UniVBE

Meanwhile, you had things like video drivers for Windows, usually basic 2D functionality:

https://www.os2museum.com/wp/windows-9x-video-minidriver-sou...

https://www.os2museum.com/wp/antique-display-driving/

In the mid-90's we started getting 3D hardware like 3Dfx and Rendition cards, they each had their proprietary interfaces.

And then finally, the 3D hardware was adopted by standards like OpenGL and DirectX.

Thanks. I agree that the 90s saw the explosion of graphic programming and graphic card techs.

I wonder if a "graphic card emulator" is a legit thing, like a CPU emulator -- or is that simply a 2d/3d/text renderer? The problem I face is, most of the legacy cards are either gone, or expensive to acquire (plus I need to buy a legacy PC), so it is not practical to write drivers for most of the legacy physical cards -- but say if I want to write a driver for a Hercules Graphic Card (admittedly it's a 80s card), can I simply write a software emulation and plug the "driver" into some place? Is it equivalent to writing the low level graphic libraries like SDL?

I would look for MiSTer FPGA PC-XT core

Thanks, I see. So it's like, instead of a softcore in Verilog, we have something similar for a GPU.

There were quite a few books and resources on the topic: Michael Abrash Zen of Graphic Programming (https://archive.org/details/zenofgraphicspro00abra), Ralph Brown interrupt list (https://ctyme.com/rbrown.htm), books like “PC Interdit” in France etc.

And online stuff as well.

Graphic programming, without a GPU, and even without a FPU, was quite interesting (here is a realtime-ish phong rendering I implemented circa 1995, without any float numbers https://m.youtube.com/watch?v=eq5hzUkOJsk).

A lot of stuff can be found online these days.

Have fun!

Thanks for sharing. I know that "Zen of Graphic programming" is mostly about "graphic programming using CPU", is this actually the same thing that a GPU driver does? It makes sense though, as GPU simply take over what the CPU was supposed to do. But I do believe there are a lot of hardware information included too. I wonder how these two parts piece together...

I created a (currently not publicly released) driver for the 1998 Sega Dreamcast's video hardware from scratch. It supports additional features over the driver in the open source homebrew OS, KallistiOS (KOS), like better render-to-texture support (the KOS driver only supports rendering to framebuffer sized textures), tile multipass (which allows for accumulation buffer style effects or soft shadows), and dynamically toggling anti-aliasing on the fly (with KOS it's fixed after init). Some screenshots of my driver can do are here: https://imgur.com/a/DyaqzZD

I used publicly available documentation (like https://www.ludd.ltu.se/~jlo/dc/ and the now defunct dcdev Yahoo Group), looked at the existing open source KOS driver, and looked at the source for Dreamcast emulators to figure out how things worked.

The GPU in the Dreamcast is a bit more complicated than PSX/PS2/GC since it doesn't accept polygons and draw them directly to the framebuffer. It's a tile-based deferred renderer, like many mobile GPUs, so it instead writes the polygons to a buffer in video RAM, then later walks through the polygons and renders the scene in tiles to an on-chip 32x32 pixel buffer, which finally gets written to RAM once.

This allows the Dreamcast to have a depth-only fillrate close to the 360 and PS3 (DC is 3.2 GPix/s vs 360/PS3 4.0 GPix/s), and it basically preforms a depth-only prepass to avoid doing texture reads for obscured texels. It can also preform per-pixel transparency sorting (order-independent transparency) with effectively no limit to the number of overlapping pixels (but the sorter is O(n^2), so a lot of overlap can become very expensive).

To get a working driver for the Dreamcast, you have to set up some structures in video RAM so that the hardware knows what polygons are in what tile. Another thing the driver needs to do is coordinate the part of the hardware that takes polygon commands and writes them to video RAM, and the part that actually does rendering. You typically double buffer the polygons, so that while the hardware is rendering one frame, user code can submit polygons in parallel for the next frame to another buffer.

My driver started as just code in "int main()" to get stuff on the screen, then I gradually separated stuff out from that into a real driver.

If anybody here wants to learn more about console graphics specifically, I think the original PlayStation is a good starting point since it's basically the earliest and simplest 3D-capable (though it would be more correct to say triangle capable, as it does not take Z coordinates at all!) GPU that still bears a vague resemblance to modern shader-based graphics pipelines. A few years ago I wrote a handful of bare metal C examples demonstrating its usage at the register level [1]; if it weren't for my lack of spare time over the last year I would have added more examples covering other parts of the console's hardware as well.

[1]: https://github.com/spicyjpeg/ps1-bare-metal

Thanks for sharing! Pikuma also has a PS1 graphic programming course that I plan to take in the future.

Most of the math and most of the problem domain are essentially the same as today. Just hardware got many magnitudes more capable. That makes all the difference.

As always: talk to hardware engineers about their hardware. Drivers are involved, because they are all about the hardware. The way that software wants to view the world doesn’t really matter there. (You probably already knew most or all of this.)

Disclaimer: Didn’t program drivers back then. But I do go deeply into that field of math.

Thanks. By saying Math I assume those are 2d/3d Maths and algos, like how to draw a line efficiently?

> The way that software wants to view the world doesn’t really matter there. (You probably already knew most or all of this.)

Actually I'm trying to understand this part. Can you please elaborate why the way that software wants to view the world doesn't matter there? My understanding is that the driver doesn't care about how the programmers are going to use the driver (could be drawing a window, drawing some fonts or drawing some polygons), but they merely need to implement certain standards?

From what I’ve seen, a lot of 90s driver work was exactly that mix of partial docs, trial-and-error with registers, and mailing some engineer at the card vendor hoping they’d admit to a bug. It wasn’t glamorous, but it’s kind of wild how much of it came down to persistence and a bit of luck

Thanks. I bet there were a lot of battle stories like what I read. Alas most of those went into history's garbage bin :/

I was even thinking about getting my hand on a few cheap physical cards (not sure which ones are cheaper), a Pentium box, and see if I can do anything -- even displaying some colors is fun enough.

Funny, I've been reviving an OS I wrote in the 90's as well as the graphics package, window manager and a couple of demo apps that I wrote to go with it. It's been a long and arduous trip down memory lane but after a couple of weeks of work I have it working under Qemu or Virtualbox. The thing that still befuddles me is how to get the COM ports to work, for some reason they really don't emulate those well in either one of those.

Yesterday evening I finally got VGA modes 16 and 18 working, so now I have 640x350 and 640x480 graphics. Major milestone, and lots of fun to get it working this far. The last time it ran was 26/Jan/1994 so it's been mothballed since then and the old hardware is long gone.

As for how it is done: you use the BIOS to switch modes, then you have some basic info about the memory map of the video mode and usually a page register where you can set which part of the memory on the graphics card is exposed in a window starting at A000:0000. Later graphics modes, such as VESA based modes used a slightly different mode switch and tended to have the option to address all of the RAM in a contiguous stretch doing away with the bankswitching, which greatly improved low level performance.

All of the graphics primitives were in software, no HW acceleration at all.

> The thing that still befuddles me is how to get the COM ports to work, for some reason they really don't emulate those well in either one of those.

What are you having trouble with for COM ports? Checking status (port + 5) and interrupt identification (port + 2) might help if you're missing interrupts?

It goes further than that: it doesn't work at all. I know quite well how to deal with hardware com ports (I've built firmware for port concentrators), it's just that the emulations do not expose working endpoints and that makes it really hard to get this going.

Qemu and Virtualbox both supply similar mechanisms that should expose the com port but I've yet to be able to input or output a single byte through an emulated com port on either. What I'll probably end up doing is first to get a hardware pass through to work and once that is reliable to move back to emulated hardware to see if there are any differences. It's funny, I expected this to be one of the easier devices but so far it has not yielded at all.

Hmmmm. I can attest to qemu com ports working for my hobby OS, which does work on hardware com ports as well. For qemu, I use the com port as the primary console. But, I just assume there's a comport at 0x3f8, I don't do detection.

I don't remember if I got virtualbox com ports to work, I had a lot of trouble with virtualbox and decided it wasn't on my support list.

This is output only, but if you don't have a better way to test if qemu comport works, you can run my kernel with

     wget https://crazierl.org/crazierl.elf
     qemu-system-i386 -display none -serial mon:stdio -kernel crazierl.elf

You should get a bunch of useless debug output, but it came through the com1. If that's helpful, the relevant code is more or less here https://github.com/russor/crazierl/blob/4e41bfa5c5bbae078a57... (init starts here, writing characters a smidge later on line 324)

Thank you, this will definitely help. I suspect that control line emulation is what is causing the problems. Detection seems to work just fine.

During that time, I had a job for a major games company doing nothing but developing Windows graphics card drivers. They were moderately complex beasts (enormously complex compared to other device drivers), but not really that huge of a thing.

The biggest effort about them was reverse-engineering certain cards. The games often used very strange video settings, and the card manufacturers had poor, sometimes no, documentation about their operation at a low level.

Thanks for sharing. I'm surprised that game companies would do that. Is it the Voodoo era or slightly earlier (S3)? Does that mean you actually ship drivers with games? What did the card manufacturers say about this?

Earlier. The standard video drivers didn't support video modes needed for more advanced video game graphics, making custom drivers necessary. The card manufacturers didn't mind. Why would they? If a hit video game supported their card, they sell more cards. Even better when they didn't have to put any substantial resources into that support.

Do you still write video drivers or is it mostly covered like with Nvidia drivers?

It was a long time ago. The trajectory of my career took me in an entirely different direction.

Thanks for sharing JohnFen.

Fun link. I never wrote GPU drivers, but it does remind me of writing my first Ethernet card driver back in the day. I felt like I had decoded the Rosetta Stone, and there was absolutely no one to talk to who understood how that felt.

You had to write your own shaders in C that would often bsod your computer and possibly brick your 3dfx voodoo card.

Less taught in CS degrees, but still definitely a huge part of computer graphics courses. Interestingly that was one of the key things you learned if you went to FullSail or DigiPen in the late 90s

Also please recall, windows development and DirectX was kind of the only game in town for PC graphics. Unreal engine made 3D indie dev realistically approachable to learn.

Interesting. I heard about FullSail and Digipen back in the day. I guess nowadays they are less relevant, because a lot of stuffs are online now.

I think shaders are usually written in a higher level language, but for drivers it is in C, right? Never written any graphic stuffs actually, but would be fun to try it out -- instant gratification and I can choose what level I want to reach -- once I can write software 3d renderer I can technically make any 3d games I want.

If this topic interests you, it's worth it to read about the rise and fall of 3dfx. Their GPUs were incredible when they came to the market and in many ways, were the genesis of the modern GPU in terms of 3D hardware acceleration.

https://en.wikipedia.org/wiki/3dfx

Thanks. Their graphic cards were very powerful but expensive back then (for me at least) so I never got the chance to use them. My first card is Geforce 3 Ti, which was quite powerful and I think it can even run the leaked Doom 3 build around 5-10 FPS.

I worked as a graphics driver programmer for Canopus in the nineties primarily customizing graphics drivers for Rendition (Verite), Nvidia (Riva128 TNT) and 3Dfx boards (Voodoo1 and Voodoo2).

Most recently I worked on a 'graphics driver' for the PS5 for Minecraft (Bedrock edition)

It's a lot to talk about but if you have any specific questions I can answer them.

Thanks! Sorry I don't have any specific questions so don't want to throw random stuffs at you. I think I'll begin by writing a software renderer and see what happens. Once I have a 3d renderer I can technically build any 3d games I wish to build (as long as it doesn't require eye candies!) so I can go from there. Drivers are still too far away from me -- and as others pointed out, it's extremely to write one without being an insider.

Not sure if this helps you, but John Carmack wrote a guide to writing GPU drivers for Quake 3 Area back in '99:

* https://web.archive.org/web/20000303225420/http://www.quake3...

Thanks! I found another earlier one for GLQuake (it's called a mini-GL driver) back in 1997:

https://github.com/id-Software/Quake/blob/master/WinQuake/3d...

Looks like it is a driver specifically for Voodoo cards to run Quake. I'm not 100$ sure if the source code is included so I'll probe a bit.

Writing graphics card drivers in the 1990s was a nightmare, and often the most complex part of an operating system, particularly ones that supported protected mode and virtual DOS machines.

Windows/386's graphics card driver was so integrated with the protected-mode kernel that it had an entirely different kernel for each graphics card. It came out of the box with support for CGA and EGA, and a later version added Herculues, VGA, and 8514/A. That was it.

By Windows 3.0, it had a more modular architecture - but this made the drivers even harder to write. A driver that would run in user space had to be written in 16-bit assembly or C. And then a secondary driver that would run in kernel space had to be written in 32-bit assembler, and that driver had to completely virtualise the device for DOS apps (and also had to provide a virtual device the aforementioned 16-bit driver could use).

Drivers for OS/2 got even more complex: the above 16-bit Windows driver was still needed. Plus a 32-bit protected-mode driver for OS/2 apps. Plus a "base" 16-bit driver for character mode when the system was booting, although most cards could use the generic VGA or SVGA driver for that. Plus the 32-bit virtualisation driver for DOS apps. Plus a specialised 16-bit Windows driver that would share the display with the 32-bit protected-mode driver so that both Windows and OS/2 apps could be on the display at the same time.

XFree86 was the usual way to support graphics cards on Linux, alongside very simple kernel-based drivers for character support - or more complex drivers for framebuffer support. If you wanted to run XFree86 on OS/2, you had to have a separate executable (just like Linux) for the specific graphics driver. So an OS/2 machine could end up having this many graphics drivers:

- Base video driver, e.g. BVHSVGA - 16-bit Windows full screen driver - 16-bit Windows seamless driver - 32-bit OS/2 Presentation Manager driver - 32-bit virtualisation driver - 32-bit XFree86 driver

NT would be yet another driver, although at least on NT it was just a single kernel space driver, and Windows 95 yet another. Windows 95 needed the same virtualisation driver as prior versions. NT didn't typically provide that, but NT was also notorious for poor support for DOS games because of that.

Some vendors actually supported all of these. You'd get a couple disks with a graphics card for GEM, DOS (VESA), OS/2 (16-bit and 32-bit), Windows 2.x, 3.x, NT 3.x, and even SCO Unix. The workload to write all these drivers was insane.

Thanks this is really great information. I never imagines that this was so messy back in the day. But considering the diversity of OS/Hardware back then, this actually makes sense. I guess it could be interesting for someone who is highly technical.

Programming Commander Keen was dramatic. Keen Dreams source code is now open

SGI ran OpenGL in hardware on their cards.

In that era I remember I had one computer with a particular Oak(?) VGA card and it ran Heretic at something like 2x the framerate of any other system. No idea why. Might have been the motherboard or the memory speed or something. Never got to the bottom of it. Total mystery.

Assuming other factors (CPU etc) are similar, possibly the card used a faster bus like VESA local bus or PCI whilst the others were plain ISA?

Pretty sure this was ISA era. So yeah, VLB would make sense. While I don't recall it being any longer than regular cards, I just checked and that was only 12mm difference. On the face of it you may have just solved the mystery, but talking to the AI most of the supposedly faster chipsets of the time like S3, Cirrus Logic, ET4000, etc. were definitely on my radar ... so I'm not sure that's the reason, because this system definitely outperformed others. My running theory was that the developer hand-tweaked their graphics code on a similar configuration system. In the absence of any hard info (honestly can't really trust my memory after ~30 years!) I guess that'll remain the guesstimate explanation.

IBM never had plans for PCs to be graphically impressive (unlike other gaming-oriented microcomputers which had hardware sprites, buffer scrolling, and various special modes), so conceptually its video cards were pretty simple: you change values in video memory, and letters or pixels on screen change accordingly. The rest was on you. Initial setup could be complex (or hidden behind Video BIOS calls to specific firmware), along with memory layout (for various reasons, including production costs), but the thing was basically just an output buffer.

There were some accelerators providing mind-blowing functions like drawing a line on screen given its start and end coordinates in hardware, or filling a rectangle with a given colour in hardware. If you used professional CAD programs where taking drawing routines away from CPU meant going from 1-2 fps to “tolerable” object movement, there was a point in paying significant sum of money for such niche device. Later, Windows strongly suggested to graphic card makers to implements most often used window drawing primitives in hardware, and offer them to the system via standard driver interface. That also was of little use to most games.

Non-standard modes and hacks to manipulate output settings to make screen scroll without redrawing the whole buffer (which was too slow), and other tricks, like changing mode parameters precisely at the moment some specific part of the image was being sent to the monitor, were sometimes possible, but they don't apply here. Doom engine games render the upper world window on CPU frame by frame, writing resulting pixels to the video memory (status bar can be partially updated, or just ignored for our case). So it's a simple stream of writes, and the difference is in how fast they are processed.

What could be different?

— Memory subsystem and address mapping. Not sure about the details, but some hardware settings could possibly tell the motherboard chipset to ignore the caching/order/coherency, and instantly return with success from any write while it was in flight. That would mean that any program that also needed to read from video memory could get incorrect or corrupted results sometimes. (Though I contradict myself here: spectres and screen wipe effect in Doom needed to read from the screen buffer.)

— System bus type, width, and rate. Smaller delays means more frames per second (if images can be calculated fast enough).

— Video memory type, rate, and timings, video chip speed and architecture, internal bus width, presence of caching buffers, and so on. Most important differences, and there are benchmarks of VGA cards in contemporary magazines and on modern retro forums.

However, the video card itself couldn't make your CPU work with twice the performance, it only could limit the amount of data that went to screen. I suspect that either the card was not very compatible, and traded simplicity and performance in certain graphical modes for bugs and slowdowns in others, or the numbers you saw were actually incorrect. For example, if video card forced 60 Hz refresh rate in canonically 70/75/85/etc Hz mode, program could start calculating nonsensical delays, insert rendering cycles that did nothing, and show crazy stats.

These were pretty proprietary I remember.

Yeah I think that was the case and still the case for many companies (nVidia). From what I briefly looked up, good thing that we can now develop drivers for virtual graphic card, and there are OSS drivers from both Intel and AMD.