[{"content":"Emulation is a fascinating area of software engineering, being able to bring to life a 30+ year old arcade machine on a modern computer is an incredibly satisfying accomplishment. Unfortunately I\u0026rsquo;ve become increasingly disillusioned with the lack of ambition shown by those in the emulation community. Whilst the rest of world moves onto cloud first, massively distributed architectures, emulation is still stuck firmly in the 20th century writing single threaded C++ of all things.\nThis project was born out of a desire to bring the best of modern design back to the the future of ancient computing history.\nSo what can the best of modern architecture bring to the emulation scene?\nHot swappable code paths allowing for in game debugging Different languages for different components Secure by default (mTLS on all function calls) Scalability Fault tolerance Cloud native design This culminated in the implementation of an 8080 microprocessor utilising a modern, containerised, microservices based architecture running on kubernetes with frontends for a CP/M test harness and a full implementation of the original Space Invaders arcade machine.\nThe full project can be found as a github organisation https://github.com/21st-century-emulation which contains ~60 individual repositories each implementing an individual microservice or providing the infrastructure. This article goes into details on the technical architecture and issues I ran into with the project.\nKey starting points to learn more are:\nA react based 8080 disassembler running on github pages - https://github.com/21st-century-emulation/disassembler-8080 The CP/M test harness used to validate the processor - https://github.com/21st-century-emulation/cpm-test-harness Just use docker-compose up --build in this repo to run the application Space Invaders UI - https://github.com/21st-century-emulation/space-invaders-ui Run locally with docker-compose up --build or use the following project to deploy into kubernetes Kubernetes Configuration \u0026amp; Deploying - https://github.com/21st-century-emulation/space-invaders-kubernetes-infrastructure Note that this presupposes that you have access to a kubernetes cluster which can handle ~200 new pods Finally, a screenshot of the emulator in action can be seen here:\nArchitectural Overview The following image describes the full archictectural model as applied to a space invaders arcade machine, the key components are then drawn out in the following sections\nCentral Fetch Execute Loop All old school emulators fall into one of two camps, either they step the CPU one instruction at a time and then catch up other components or they step each component (including the CPU) one cycle at a time. The 8080 as played in a space invaders cabinet gains nothing from being emulated with cycle level accuracy so this emulator adheres to the former design. The Fetch Execute Loop service is the service which then performs that core loop and is broadly shaped as follows\nwhile true: Call microservice to check if interrupts should occur If so then run RST x instruction Get next instruction from memory bus microservice Call corresponding opcode microservice That\u0026rsquo;s it. In order to actually drive this microservice we also provide /api/v1/start and /api/v1/state endpoints which correspondingly trigger a new instance of the CPU to run and get the status of the currently running CPU.\nOpcode microservices Every opcode corresponds to a microservice which must provide a POST api at /api/v1/execute taking a JSON body shaped as follows:\n{ \u0026#34;id\u0026#34;: \u0026#34;uuid\u0026#34;, \u0026#34;opcode\u0026#34;: 123, // Current opcode used to disambiguate calls to e.g. MOV (MOV B,C or MOV B,D) \u0026#34;state\u0026#34;: { \u0026#34;a\u0026#34;: 0, \u0026#34;b\u0026#34;: 0, \u0026#34;c\u0026#34;: 0, \u0026#34;d\u0026#34;: 0, \u0026#34;e\u0026#34;: 0, \u0026#34;h\u0026#34;: 0, \u0026#34;l\u0026#34;: 0, \u0026#34;flags\u0026#34;: { \u0026#34;sign\u0026#34;: false, \u0026#34;zero\u0026#34;: false, \u0026#34;auxCarry\u0026#34;: false, \u0026#34;parity\u0026#34;: false, \u0026#34;carry\u0026#34;: false, }, \u0026#34;programCounter\u0026#34;: 100, \u0026#34;stackPointer\u0026#34;: 1000, \u0026#34;cyclesTaken\u0026#34;: 2000, \u0026#34;interruptsEnabled\u0026#34;: false, } } Memory bus The memory bus serves to provide the stateful storage for the service and must expose 4 routes to the other services:\n/api/v1/readByte?id=${cpuId}\u0026amp;address=${u16} - Read a single byte from the address passed in /api/v1/writeByte?id=${cpuId}\u0026amp;address=${u16}\u0026amp;value=${u8} - Write a single byte to the address passed in /api/v1/readRange?id=${cpuId}\u0026amp;address=${u16}\u0026amp;length=${u16} - Read at most length bytes starting at address (to get e.g. the 3 bytes that correspond to an instruction) /api/v1/initialise?id=${cpuId} - POST takes a base64 encoded string as body and uses that to initialise the memory bus for the cpu id passed in There\u0026rsquo;s a simple \u0026amp; fast implementation written in rust with a high tech in memory database provided at https://github.com/21st-century-emulation/memory-bus-8080. Alternative implementations utilising persistent storage are left as an exercise for the reader. A blockchain based backend is probably the best solution to this problem.\nInterrupt service When running the fetch execute loop service you can optionally provide (via an environment variable) the url to an interrupt check service which will be called before every opcode is executed. This API must take the same JSON body as the opcode microservices and will return an optional value which indicates which RST opcode is to be taken (or none if no interrupt is to be fired).\nDeployment architecture Whilst the application can be run locally using docker-compose, no self-respecting cloud solutions architect would be satisfied with the risks inherent in having everything pinned to a single machine. Consequently this project also delivers a helm chart which can be found here.\nGiven that repository and a suitably large kubernetes cluster (note: we strongly recommend choosing a top tier cloud provider like IBM for this), all components can be installed by simply running ./install.sh.\nThe kubernetes architecture is outlined in https://github.com/21st-century-emulation/space-invaders-kubernetes-infrastructure/blob/main/README.md but a diagram is provided here for brevity:\nPerformance As with all modern design it\u0026rsquo;s crucial to adhere to the model of \u0026ldquo;make it work then make it fast\u0026rdquo; and that\u0026rsquo;s something that this project really takes to heart. In 1974 when the 8080 was released it achieved a staggering 2MHz. Our new modern, containerised, cloud first design doesn\u0026rsquo;t quite achieve that in it\u0026rsquo;s initial iteration. As can be seen from the screenshot above, space invaders as deployed onto an AKS cluster runs at ~1KHz which gives us ample time for debugging but does make actually playing it slightly difficult.\nHowever, now that the application works we can look at optimising it, the following are clear future directions for it to go in:\nRewrite more things in rust. As we can see in the image below, a significant portion of the total CPU time was spent running LXI \u0026amp; POP opcodes. This is quite understandable because LXI is written in Java/Spring and POP is written in Scala/Play. Both are clearly orders of magnitude slower than all the other languages in play here. JSON -\u0026gt; Avro/Protobuf. JSON serialisation/deserialisation is known to be too slow for modern applications, using a better binary packed format will obviously increase performance Pipelining \u0026amp; speculative execution. A minor speed boost can be achieved by simply pipelining up to the next N instructions and invalidating the pipeline on any instruction which changes the program counter. This is particularly excellent because it brings modern CPU design back to the 8080! Since all operations internally are async and wait on IO we can trivially execute multiple instructions in parallel, a further enhancement would therefore be to speculatively execute instructions and rollback if the execution of a previous one would have affected the result. Memory caches Having to access the memory bus each time is slow, by noting which instructions can affect memory we are able to act like a modern VM and cache memory until a write happens at which point we invalidate the cache and continue. See the below image showcasing the amount of requests made to /api/v1/readRange form the fetch execute loop (which uses that API to get the next instruction). Implementation Details One of the many beautiful things about a microservice architecture is that, because function calls are now HTTP over TCP, we\u0026rsquo;re no longer limited to a single language in our environment. That allows us to really leverage the best that modern http api design has to offer.\nThe following table outlines the language choice for each opcode, as you can see, this allows to gain the benefits of Rusts safe integer arithmetic operations whilst falling back to the security of Deno for important operations like CALL \u0026amp; RET.\nOpcode Language Description Runtime image size Performance (avg latency) MOV Swift Moves data from one register to another 257MB 4.68ms MVI Javascript Puts 8 bits into register, or memory 118MB 3.43ms LDA VB Puts 8 bits at location Addr into A Register 206MB 4.56ms STA C# Stores 8 bits at location Addr 206MB 4.61ms LDAX Typescript Loads A register with 8 bits from location in BC or DE 365MB 6.22ms STAX Python Stores A register at location in BC or DE 59MB 5.24ms LHLD Ruby Loads HL register with 16 bits found at Addr and Addr+1 898MB! 13.63ms SHLD Perl Stores HL register contents at Addr and Addr+1 930MB! 12.68ms LXI Java + Spring Loads 16 bits into B,D,H, or SP 415MB 6.84ms PUSH Lua Puts 16 bits of BP onto stack SP=SP-2 385MB 4.42ms POP Scala + Play Takes top of stack, puts it in RP SP=SP+2 761MB 13.99ms XTHL D Exchanges HL with top of stack 156MB 26.54ms SPHL F# Puts contents of HL into SP (stack pointer) 114MB 3.25ms PCHL Kotlin Puts contents of HL into PC (program counter) [=JMP (HL)] 445MB 7.61ms XCHG C++ Exchanges HL and DE 514MB 2.16ms ADD Rust Add accumulator and register/(HL) 123MB 1.95ms ADC Rust Add accumulator and register/(HL) (with carry) 123MB 2.00ms ADI Rust Add accumulator and immediate 123MB 2.16ms ACI Rust Add accumulator and immediate (with carry) 123MB 2.22ms SUB Rust Sub accumulator and register/(HL) 123MB 1.95ms SBB Rust Sub accumulator and register/(HL) (with borrow) 123MB 1.70ms SUI Rust Sub accumulator and immediate 123MB 2.15ms SBI Rust Sub accumulator and immediate (with carry) 123MB 1.91ms ANA Rust And accumulator and register/(HL) 123MB 2.68ms ANI Rust And accumulator and immediate 123MB 1.93ms XRA Rust Xor accumulator and register/(HL) 123MB 1.70ms XRI Rust Xor accumulator and immediate 123MB 1.57ms ORA Rust Nim Or accumulator and register/(HL) 74MB 11.36ms ORI Rust Or accumulator and immediate 123MB 1.40ms DAA Rust Decimal adjust accumulator 123MB 2.26ms CMP Rust Compare accumulator and register/(HL) 123MB 1.70ms CPI Rust Compare accumulator and immediate 123MB 1.90ms DAD PHP Adds contents of register RP to contents of HL register 430MB 17.2 ms INR Crystal Increments register 23MB 1.98ms DCR Crystal Decrements register 23MB 2.06ms INX Crystal Increments register pair 23MB 2.01ms DCX Crystal Decrements register pair 23MB 1.99ms JMP Powershell Unconditional Jump to location Addr 294MB 6.51ms CALL Deno Unconditional Subroutine call to location Addr 154MB 6.04ms RET Deno Unconditional return from subroutine 154MB 6.43ms RLC Go Rotate left carry 6MB 2.28ms RRC Go Rotate right carry 6MB 2.19ms RAL Go Rotate left accumulator 6MB 2.39ms RAR Go Rotate right accumulator 6MB 2.29ms IN Data from Port placed in A register OUT Data from A register placed in Port CMC Haskell Complement Carry Flag 90MB 2.50ms CMA Haskell Complement A register 90MB 2.54ms STC Haskell Set Carry Flag = 1 90MB 2.52ms HLT Halt CPU and wait for interrupt NOOP C No operation 70MB 1.89ms DI Dart Disable Interrupts 79MB 2.37ms EI Dart Enable Interrupts 79MB 2.21ms RST Deno Call interrupt vector 154MB 7.34ms Nim was a bit late to the party so only got one opcode, and it still managed to be slow anyway.\nCode details According to SCC this project cost $1M to make, which is probably several orders of magnitude less than Google will pay for it. Like any true modern application it also consists of significantly more JSON/YAML than code.\n─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code Complexity ─────────────────────────────────────────────────────────────────────────────── YAML 144 6873 859 396 5618 0 Dockerfile 112 2007 505 353 1149 248 JSON 106 15383 240 0 15143 0 Shell 64 2448 369 195 1884 257 Plain Text 59 404 171 0 233 0 Docker ignore 56 295 0 0 295 0 Markdown 56 545 165 0 380 0 gitignore 56 847 129 178 540 0 C# 35 1803 198 10 1595 51 TypeScript 22 1335 116 30 1189 275 Rust 18 1825 241 7 1577 79 TOML 18 245 20 0 225 0 Java 7 306 66 0 240 1 Haskell 6 207 24 0 183 0 Visual Basic 6 119 24 0 95 0 MSBuild 5 53 13 0 40 0 Crystal 4 330 68 4 258 5 Go 4 255 33 0 222 6 JavaScript 4 147 8 1 138 5 License 4 84 16 0 68 0 PHP 4 147 21 43 83 2 Swift 4 283 15 4 264 1 C++ 3 32 4 0 28 0 Emacs Lisp 3 12 0 0 12 0 Scala 3 112 15 0 97 6 XML 3 97 13 1 83 0 C Header 2 24 2 0 22 0 CSS 2 44 6 0 38 0 Dart 2 58 6 0 52 18 HTML 2 361 1 0 360 0 Lua 2 65 8 0 57 15 Properties File 2 2 1 0 1 0 C 1 63 8 0 55 18 CMake 1 68 10 15 43 4 D 1 71 6 2 63 14 F# 1 88 11 3 74 0 Gemfile 1 9 4 0 5 0 Gradle 1 32 4 0 28 0 Kotlin 1 40 9 0 31 0 Makefile 1 16 4 0 12 0 Nim 1 82 9 0 73 2 Perl 1 49 6 3 40 4 Powershell 1 78 9 0 69 10 Python 1 37 6 0 31 1 Ruby 1 28 6 0 22 0 SVG 1 3 0 0 3 0 TypeScript Typings 1 1 0 1 0 0 ─────────────────────────────────────────────────────────────────────────────── Total 833 37413 3449 1246 32718 1022 ─────────────────────────────────────────────────────────────────────────────── Estimated Cost to Develop (organic) $1,052,256 Estimated Schedule Effort (organic) 14.022330 months Estimated People Required (organic) 6.666796 ─────────────────────────────────────────────────────────────────────────────── Processed 1473797 bytes, 1.474 megabytes (SI) ─────────────────────────────────────────────────────────────────────────────── Issues Naturally I ran into a number of new issues with this approach. I\u0026rsquo;ve listed some of those below to give a flavour for the types of problems with this architecture:\nGithub actions\u0026hellip; Random timeouts logging in to ghcr.io, random timeouts pushing images. Generally just spontaneous errors of all sorts. This really drove home how fun it is managing the development toolchain for a microservice architecture Haskell compiler images Oh boy. Haskell won the award for my least favourite development environment solely off the back of the absurd 3.5GB SDK image! That was sufficiently large that it was impossible to build the haskell based services in CI without fine tuning the image down to \u0026lt; 3.4GB (github actions limits) Intermittent AKS networking errors Whilst it achieved ~4 9s availability across all microservices, there were spontaneous 504s between microservices in the AKS implementation. On the plus side, because we\u0026rsquo;re using linkerd as a service mesh to give us secure microservice TCP connections we can also just leverage it\u0026rsquo;s retry behavior and forget about the problem! Exactly like a modern architecture! DNS caching (or not) Only node.js of all the languages used had issues where it would hammer the DNS server on literally every HTTP request, eventually DNS told it to piss off and the next request broke #justnodethings Logging at scale I initially set up Loki as the logging backend because it\u0026rsquo;s new and therefore good, but found that the C# libraries for Loki would occasionally send requests out of order and that in the end Loki would just give up and stop accepting logs - fortunately fluentd is still very much in the spirit of this project and really pins the project down to kubernetes so it was obviously the best decision all along Orchestrating changes across services Strangely, having ~50 repositories to manage was marginally harder than having 1. Making a change to (for example) add an interruptsEnabled flag to the CPU needed to be orchestrated across all microservices. Fortunately I\u0026rsquo;m quite good at writing disgusting bash scripts like any self respecting devops engineer. Is this actually possible? Alright, if you\u0026rsquo;ve got this far I\u0026rsquo;m sure you\u0026rsquo;ve realised that the whole project is something of a joke. That said it is also an interesting intellectual exercise to consider whether it\u0026rsquo;s remotely possible to achieve \u0026gt;=2MHz with the architecture delivered.\nThe starting point is that to achieve 2MHz we must deliver 1 instruction every 2μs\n2MHz = 2,000,000 cycles per second Each instruction is at 4-17 cycles so we need to manage at worst 2,000,000 / 4 = 500,000 instructions per second. That gives 1/500,000 seconds = ~2μs per operation. As it was written there are 3 HTTP calls per instruction, one to fetch the operation, one to execute it and one to check for interrupts.\nAssuming for sake of argument that we do the following optimisations:\nMake interrupt checks off the hot path Cache all ROM in the fetch execute service and assume applications only execute from ROM (true for space invaders) This takes us to ~1 instruction per operation Change from JSON w/ UTF8 encoding to sending a byte packed array of values to represent the CPU Drives the request size down to \u0026lt;256 bytes and eliminates all serialization/deserialization costs (just have a struct pointer pointing at the array) Then we can get to a good starting point of exactly 1 round trip to/from each opcode. So what\u0026rsquo;s the minimal cost for a roundtrip across network?\nThis answer (https://quant.stackexchange.com/questions/17620/what-is-the-current-lowest-possible-latency-for-tcp-communication) from 2015 benchmarks loopback device latency at ~2μs if the request size can be kept down to \u0026lt;=256 bytes.\nAssuming that person knows what they\u0026rsquo;re talking about then the immediate answer is a straight no. You\u0026rsquo;ll never achieve the required latency across a network (particularly a dodgy cloud data center network).\nBut let\u0026rsquo;s not give up quite yet. We\u0026rsquo;re not miles away from the performance required so we can look for 2 times speed ups.\nSome thoughts on methods to get that last 2 times speed up:\nFire and forget memory writes A memory write is almost never read immediately, so just chuck it at the bus and don\u0026rsquo;t bother blocking until it\u0026rsquo;s written. Maybe you\u0026rsquo;ll lose some writes? That\u0026rsquo;s fine. Very mongo. fsync is for boring c coders and modern developers aren\u0026rsquo;t supposed to need to know about nasty complex things like the CAP theorem anyway. Presumably kubernetes will solve that for us. We can execute multiple operations in parallel and only validate the correctness of their results later. This would clearly speed up operations like memset\u0026rsquo;s which are done with simple MVI (HL) d8 -\u0026gt; DCX HL -\u0026gt; JNZ type algorithms where each grouping can be executed in parallel If each opcode was capable of knowing the next instruction then we could avoid the second half of each round trip and not travel back to the fetch execute loop until the stream of instructions has run out This is basically a guaranteed 2 times speed up Conclusion? I think it might be possible under some ideal situations assuming ~no network latency but I\u0026rsquo;ve no intention of spending any more time thinking about it!\n","permalink":"https://blog.davetcode.co.uk/post/21st-century-emulator/","summary":"What happens when you apply 21st century application design to the emulation of old arcade machines? How much can we improve the emulation community?","title":"Bringing emulation into the 21st century"},{"content":"I\u0026rsquo;ve spent a lot of spare time over the last few years playing with emulating old computer architectures. I also spend my day job mentoring (and even occasionally writing) C#. This project combines those two areas of interest and asks the question: \u0026ldquo;Is it possible to recompile 8080 assembly into CLR IL\u0026rdquo;\nDid it work? Yes! I have successfully built a fully functional Space Invaders emulator with the 8080 cpu core purely implemented in IL.\nThis blog post covers some of the details of what 8080 assembly looks like, what IL looks like, the process of mapping one to the other and also a fair bit of details about the complications involved.\n8080 Architecture To start we need an understand of what an 8080 actually is. Interally the 8080 is a processor with the 7 8 bit registers, 5 flags, a 16 bit stack pointer, and a 16 bit program counter.\nRegisters Flags A (accumulator) Carry flag (did the last result carry 255 + 1 = 0 w/carry 0-1=255 w/carry) B Sign flag (was the result of the last operation twos complement negative?) C Zero flag (was the result of the last operation zero?) D Parity flag (did the last result have even parity or odd parity?) E Aux Carry flag (c.f. https://en.wikipedia.org/wiki/Adjust_flag) H L Externally, treated as a black box, it\u0026rsquo;s quite a simple architecture with the following inputs \u0026amp; outputs:\nA 16 bit address bus which can have lines hooked up to anything from raw memory, to memory mapped IO devices. That is, whilst the CPU can only reference addresses from 0x0000-0xFFFF there\u0026rsquo;s no guarantees whatsoever that addresses refer to R/W memory. One could be hooked up to an LED and writing to it changes the color of the LED, the next could not be wired up at all etc. Up to 255 IN/OUT ports which can be hooked up to any devices the designers selected (sound chips, serial ports, dedicated hardware for mathematical operations, etc) A single interrupt line from which the external architecture can say \u0026ldquo;jump to start executing at address X\u0026rdquo; where X is one of a pre-defined set of 8 addresses Note: It\u0026rsquo;s obviously a lot more complicated than that in detail, this high level view is sufficient for our emulation!\n8080 Assembly With that context in mind we can look at what a simple piece of 8080 assembly code looks like both in textual form and in assembled bytes. The 8080 provides 255 different operations that it can perform. These generally characterise as:\n8 bit arithmetic e.g. ADI $ =\u0026gt; register A = A + $ (the next byte) and set all the flags CMP B =\u0026gt; Check value of A - B and set all flags (but don\u0026rsquo;t actually change A) 16 bit arithmetic e.g. INX B =\u0026gt; BC = BC + 1 8 bit load/store e.g. LDA a16 =\u0026gt; Get whatever is at memory address a16 and put it in A MVI E,d8 =\u0026gt; Put the next byte into E Jumps/Calls e.g. JZ $addr =\u0026gt; Jump immediately to the fixed 16 bit address $addr if and only if the zero flag is set CNC $addr =\u0026gt; Push next instruction to stack and then jump to $addr if and only if the carry flag is not set RET =\u0026gt; Return to the address referred to by the top two bytes of the stack PCHL =\u0026gt; Set the program counter to the value of the HL register pair (dynamic jump) Misc e.g. NOP =\u0026gt; Do nothing HLT =\u0026gt; Stop cpu Code like this will get assembled from something like (taken from https://en.wikipedia.org/wiki/Intel_8080#Example_code):\nmov a,b ;Copy register B to register A ora c ;Bitwise OR of A and C into register A rz ;Return if the zero-flag is set high. loop: ldax d ;Load A from the address pointed by DE mov m,a ;Store A into the address pointed by HL inx d ;Increment DE inx h ;Increment HL dcx b ;Decrement BC (does not affect Flags) mov a,b ;Copy B to A (so as to compare BC with zero) ora c ;A = A | C (set zero) jnz loop ;Jump to \u0026#39;loop:\u0026#39; if the zero-flag is not set. ret ;Return to the bytes 78 B1 C8 1A 77 13 23 0B 78 B1 C2 03 10 C9 where the byte 0x78 literally means \u0026ldquo;MOV A,B\u0026rdquo; or \u0026ldquo;Move the value in register B into A\u0026rdquo;.\nSo an emulators job is to take that stream of bytes and pretend to be the cpu by performing each operation in turn. A typical basic emulator of old 8 bit architectures would look like this:\nwhile (true) { byte instruction = _memoryBus.ReadByte(_programCounter); _programCounter++; switch (instruction) { ... case 0x78: // MOV A, B _registers.A = _registers.B; break; ... } } A lot less hard than it sounds! Some opcodes are naturally a bit more complex to write than that but that\u0026rsquo;s the principle.\nCLR IL background Ok, so that\u0026rsquo;s the basics of an 8080, now lets take a look at the basic building blocks of the CLR IL and see where we can draw comparisons.\nThere are two major steps to get any dotnet application to actually run.\nCompilation Runtime JIT When you write C# and do dotnet build the output is normally an intermediary format which is interpreted by the runtime. So what does that intermediate format look like? In steps dotnet-ildasm\nConsider the following class\nclass Emulator { private byte H; private byte L; private ushort HL { get =\u0026gt; (ushort)((H \u0026lt;\u0026lt; 8) | L); set { L = (byte)value; H = (byte)(value \u0026gt;\u0026gt; 8); } } } Lets build it with dotnet build and then run dotnet ildasm assembly.dll to see what we get (I\u0026rsquo;ve removed everything except the get_HL method)\n.method private hidebysig specialname instance default uint16 get_HL() cil managed { // Method begins at Relative Virtual Address (RVA) 0x2050 // Code size 17 (0x11) .maxstack 8 IL_0000: ldarg.0 IL_0001: ldfld byte TestILAddFields.Emulator::H IL_0006: ldc.i4.8 IL_0007: shl IL_0008: ldarg.0 IL_0009: ldfld byte TestILAddFields.Emulator::L IL_000e: or IL_000f: conv.u2 IL_0010: ret } // End of method System.UInt16 TestILAddFields.Emulator::get_HL() I reckon if you squint a bit then that doesn\u0026rsquo;t look miles different from the 8080 assembly in the previous section does it. Consider what it does with some commenting:\nIL_0000: ldarg.0 // Load \u0026#34;this\u0026#34; so that the next variable can be loaded from the right class IL_0001: ldfld byte TestILAddFields.Emulator::H // Load the value of H onto the stack IL_0006: ldc.i4.8 // Load the value \u0026#34;8\u0026#34; onto the stack IL_0007: shl // Take H back off the stack, take 8 off the stack, shift H left by 8 (H \u0026lt;\u0026lt; 8 in the csharp code above) and then put the result back on the stack So what are the big differences in architecture between this model and the 8080? Well, the crucial one is that the CLR uses a stack based execution model instead of registers. That is, if I want to add two numbers in 8080, I put one into A, one into another register (say C) and then call ADD C which puts the result back in A. If I want to add two numbers in CLR I push both onto the stack, call OpCodes.Add and then the result is back on the stack.\nWhat else is different?\nThe CLR doesn\u0026rsquo;t have the concept of flags, whilst there are Beq (branch equal), Blt (branch less than) etc style operations there\u0026rsquo;s no concept of carrying around persistent state about whether the last operation resulted in zero. The CLR doesn\u0026rsquo;t have the notion of a program counter which can be directly affected by operations, that is, you can\u0026rsquo;t dynamically at runtime say \u0026ldquo;now run instruction at address X\u0026rdquo; (where X is dynamically calculated) Implementation OK, that\u0026rsquo;s a crap ton of background to make sure we\u0026rsquo;re all talking the same language. How did I take all of that and create an emulator?\nStep 1 - Dynamically construct a class which looks like the 8080 CPU Here I have a single static function CreateEmulator(byte[] rom) which returns a dynamically created type shaped like this:\nclass Cpu8080 { byte A, B, C, D, E, H, L; ushort StackPointer; bool Carry, Sign, Zero, Parity, AuxCarry; Run() {} } Internally that\u0026rsquo;s done by making use of AssemblyBuilder, TypeBuilder etc.\nWe\u0026rsquo;ll deal with how the Run method was generated in a bit.\nStep 2 - Hook in all the external interfaces What about memory access? Or access to arbitrary IN/OUT ports? Or providing hooks back so that the overarching computer can decide to render the screen?\nFor this I pass in interfaces to the various architectural interfaces. So there\u0026rsquo;s an IMemoryBus which provides a ReadByte and WriteByte function that the CPU can call but is implemented by the calling code. Likewise for IN/OUT theres an IIOHandler with In \u0026amp; Out functions.\nStep 3 - Recompile the ROM to generate the Run function The meat of the smarts here are in how we take the byte[] that corresponds to the ROM and turn it into C# IL. That process works as follows:\nCreate an ILGenerator for the Run method Define labels for every instruction (so 0xFFFF labels corresponding to all the memory addresses) Starting at 0x0000 looping until we\u0026rsquo;ve checked all possible addresses Decode the operation (is it NOP, LXI, JMP etc) Emit the label corresponding to the current program counter (so we can jump to it as needed) Emit equivalent IL for what that instruction is trying to do Go to the next instruction (might not be adjacent if the operation consumed more than 1 byte) Start the process in 3 again from the first \u0026ldquo;not yet seen\u0026rdquo; address until we\u0026rsquo;ve visited all addresses Example (OUT instruction)\nmethodIL.Emit(OpCodes.Ldarg_0); methodIL.Emit(OpCodes.Ldfld, internals.IOHandlerField); // This is the IIOHandler field referring to some external object methodIL.EmitLd8Immediate( _port); methodIL.Emit(OpCodes.Ldarg_0); methodIL.Emit(OpCodes.Ldfld, internals.A); methodIL.Emit(OpCodes.Callvirt, internals.IOHandlerField.FieldType.GetMethod(\u0026#34;Out\u0026#34;)!); // Call Out(port, A) consuming port and A from the execution stack So the output of this is a freaking massive function with 0xFFFF labels, and on average ~10 IL instructions per 8080 instruction.\nStep 4 - Solve the problem of dynamic jumps The above process is entirely sufficient to execute an 8080 in CLR IL EXCEPT for one major problem. Dynamic (runtime) jumps. This issue is mostly easily seen with the PCHL instruction. PCHL literally means \u0026ldquo;put the register pair HL into the program counter\u0026rdquo;, or \u0026ldquo;jump to the address at HL\u0026rdquo;.\nThere\u0026rsquo;s no way to statically analyse a ROM and know where that will jump to. So we need to have some way, at runtime, of saying \u0026ldquo;Jump to the IL label with value = X\u0026rdquo;.\nI implemented this with a linear jump table placed at the front of the function. In pseudo code that is roughly:\nSet destination address variable to HL OpCodes.Br JumpTableStart The jump table (which is statically generated) then looks like this:\nLoad destination address Compare to 0x0000 OpCodes.Brtrue 0x0000 For every single possible opcode (0xFFFF of them). That\u0026rsquo;s a pretty disgusting solution but I wasn\u0026rsquo;t able to come up with anything better. Presumably an OpCodes.Switch would be better but it still suffers from the same issue of size bloat. Technically this could be implemented by a binary chop through the 0xFFFF space reducing from O(n) to O(log(n)) complexity for each dynamic jump but that would increase the program size so I decided not to.\nStep 5 - Solve interrupts The biggest hurdle to overcome in any static/jit compilation solution to emulation is synchronisation with external peripherals. For some systems that\u0026rsquo;s particularly painful (to get full NES rom compatibility you\u0026rsquo;d need to synchronise every memory read/write!) but for the space invaders ROM we can get away with only synchronising on each instruction.\nIn an architectural sense what happens here is that any connected device can \u0026ldquo;notify\u0026rdquo; the CPU on the interrupt line and the CPU will (before the next instruction), clear that line and JMP to one of the RST addresses (0x00, 0x08, 0x10 etc) where it will continue operation.\nTo implement that in this recompilation we need to poll for that possible interrupt in between each instruction.\nmethodIL.MarkLabel(cpuInternal.ProgramLabels[programCounter]); // Check for interrupts interruptUtils.PostInstructionEmit(methodIL, cpuInternal, (ushort)programCounter); So we delegate to the owning computer architecture, asking it to inject code into the method which will check for and handle interrupts.\nIn the case of space invaders, this piece of code will count down the number of cpu cycles and fire a half screen interrupt after 17066 cycles and a full screen interrupt after another 17066 cycles. That code is almost certainly the most complex of all the IL I emit and it needs to appear before every single entry instruction!\nIssues On the whole this wasn\u0026rsquo;t actually that hard to implement. The single biggest issue is that you get exactly no help from the runtime at all if you\u0026rsquo;ve written bad IL. I definitely got sick of seeing InvalidProgramException by the end of the project!\nOtherwise, the space invaders system is not hard to implement, the extra shift register hardware is trivial, the interrupts are regular and easy to understand and crucially the rendering pipeline is all of 20 lines of code instead of the exceptionally complex Gameboy/Nes style PPU.\nPerformance Obviously there\u0026rsquo;s absolutely NO point in doing this from a performance point of view. Even python running on cpython can cope with emulating an 8080 at full speed, however it\u0026rsquo;s interesting to see what sort of performance it achieved.\nAveraging the time taken for all but the first frame to render resulted in 382fps on a fairly powerful laptop\u0026hellip;which is honestly not very impressive. You\u0026rsquo;d expect to be able to get well over 1000fps running an old system like this.\nThe more interesting performance note was that the actual invocation of the Run method delegate took over 1s before actually running any code. That\u0026rsquo;s over 1s for the runtime to JIT my overly long function! That time notably increased when adding the interrupt handling so size of function (or possibly number of label markers) directly affects the JIT speed.\n","permalink":"https://blog.davetcode.co.uk/post/jit-8080/","summary":"Is it possible to recompile 8080 assembly into CLR IL? Can that be extended to deliver the Space Invaders arcade machine as a dynamically recompiled CLR binary?","title":"Experiments in 8080 static/jit compilation"},{"content":"Over a period of 2 months in 2020 I intermittently worked on a NES emulator in rust to build a proper opinion of the language (and because it\u0026rsquo;s fun). On the whole it was quite successful, https://github.com/DaveTCode/nes-emulator-rust now exists, has coverage of maybe 80% of the total ROM dumps out there, and passes almost all of the test rom suite.\nAll achieved with no unsafe, no Rc/Refcell, minimal heap allocations (we don\u0026rsquo;t know how big a Rom is at compile time, it might be 8KB or 256KB) and a small handful of external crates.\nThis blog post is a write up of my feelings around rust as a language but probably provides some insights into writing an emulator along the way. I\u0026rsquo;m not claiming any level of expertise as a rust developer and only marginally more as an emulator developer!\nTL;DR This is quite long. Broad summary was that I was pleasantly surprised with all the bits surrounding the language (tooling, IDE support etc) and I liked the WIBNI style language features I found but that my key takeaway was that moreso than other languages, you need more up front knowledge both of rust and the domain you\u0026rsquo;re working in to produce high quality code. I don\u0026rsquo;t know if that\u0026rsquo;s controversial, you\u0026rsquo;ll have to read on if you think I\u0026rsquo;m wrong!\nLanguage The good There was an awful lot to like about rust as a core language, I\u0026rsquo;ve listed some of those that really stood out here but on the whole I almost never felt let down by writing code that would be more readable/beautiful if I only had feature X. The only really notable example is that https://doc.rust-lang.org/beta/unstable-book/language-features/generators.html in stable would have allowed me to write a CPU state machine with the language managing state instead of me doing it. That\u0026rsquo;s fine, it\u0026rsquo;s coming along in the future.\nwrapping_add \u0026amp; saturating_add + panics when you over/underflow in non-release builds was an absolute life saver. Emulators of 8 bit computers spend a ton of time working with 8 bit integers (and 16 bit for some of the addresses), being forced to behave myself about when I want to wrap, saturate or not was really valuable. The extra overhead is a problem in an emulator but knowing I\u0026rsquo;ll get a panic when I deliberately revert to explicit addition if I\u0026rsquo;m wrong about my invariants caught at least half a dozen bugs e.g. https://github.com/DaveTCode/nes-emulator-rust/blob/master/src/cpu/mod.rs#L1031 is fine because I know that a DMA address will only ever run for 0xFF bytes, whereas https://github.com/DaveTCode/nes-emulator-rust/blob/master/src/cpu/mod.rs#L135 increments the PC by 1, which CAN wrap around the address space and so wrapping_add.\nMatch statements are as mature as anything I\u0026rsquo;ve seen and I massively abused them throughout\nThe below is just kinda gorgeous code compared to the if statement equivalent. Nothing particularly new to me coming from C# but most other modern languages don\u0026rsquo;t have them and they are fuller featured here anyway, match address { 0x0000..=0x1FFF =\u0026gt; self.ram[(address \u0026amp; 0x7FF) as usize], 0x2000..=0x2007 =\u0026gt; self.ppu.read_register(address), 0x2008..=0x3FFF =\u0026gt; self.ppu.read_register((address \u0026amp; 7) + 0x2000), 0x4000..=0x4013 | 0x4015 =\u0026gt; self.apu.read_byte(address), // APU registers 0x4014 =\u0026gt; 0x00, // TODO - Is this correct? We read 0 on the DMA register? 0x4016..=0x4017 =\u0026gt; self.io.read_byte(address), // Controller registers 0x4018..=0x401F =\u0026gt; 0x00, // TODO - Unused APU \u0026amp; IO registers 0x4020..=0xFFFF =\u0026gt; self.prg_address_bus.read_byte(address), } I was a bit disappointed with if let matching and the inability to chain if let Some(x) \u0026amp;\u0026amp; x == something, nice to have a couple of little pointless things to grumble about though. Number formatting is honestly necessary in an emulator, deconstructing a value into bit fields with _ spaced sections is something I would get really annoyed without\nself.internal_registers.temp_vram_addr \u0026amp; 0b0000_0100_0001_1111 Macros seem really cool but I\u0026rsquo;m still scared of them after C. I made one standard (and quite nice) use in https://github.com/DaveTCode/nes-emulator-rust/blob/master/tests/test_roms.rs to construct parameterised tests but I feel like if I really got them then I could rewrite the CPU using macros instead of a state machine and the end result would have been more readable and most likely notably more performant.\nProper discriminated unions are something I\u0026rsquo;ve harped on at people about before (C# doesn\u0026rsquo;t have them although F# does), I made pretty solid use of them in various bits of this emulator. Particularly for building a sprite fetch state machine but there\u0026rsquo;s lots of other use cases as well:\n#[derive(Debug, Copy, Clone)] enum SpriteEvaluation { ReadY, WriteY { y: u8 }, ReadByte { count: u8 }, WriteByte { count: u8, value: u8 }, Completed, } The unsure Traits \u0026amp; structs vs interfaces \u0026amp; objects never really clicked with me but I think it\u0026rsquo;s probably just a case of a paradigm shift that I haven\u0026rsquo;t fully acclimatised to yet. The key example in a NES emulator is mappers. A front loading NES has a cartridge slot that you plug the game cartridge into, there are various address lines attached to it but basically it breaks down to two interfaces:\nA 16bit wide address bus to the CPU A 14bit wide addess bus to the PPU (side note, wouldn\u0026rsquo;t it be great if languages could allow for arbitrary width integers with corresponding wrapping_add etc) A natural way to model a cartridge in an inheritance \u0026amp; interface based language might be:\ninterface CpuAddressBus { read_byte(address: u16, cycle: CpuCycle) -\u0026gt; u8; write_byte(address: u16, value: u8, cycle: CpuCycle); } abstract class CpuCartridge : CpuAddressBus { rom: Vec\u0026lt;u8\u0026gt;; read_byte(address) ... blah } Then crucially, when you find that there\u0026rsquo;s a mapper which blocks writes on consecutive cycles you can extend that base class with an extra field tracking last_write_cycle using inheritance and overwrite the write_byte function. My implementation of this in rust required a trait for the CpuAddressBus, a base structure and then composition to put that base structure into MyMapper, that means you awkwardly end up writing passthrough functions all over the place\nfn read_byte(\u0026amp;mut self, address: u16, _: u32) -\u0026gt; u8 { self.base.read_byte(address) } which feels a bit crap.\nThe whole mapper section of my code base is probably one of the key ones where I\u0026rsquo;d like a code review from an experienced rustacean, I\u0026rsquo;m not at all committed to this being a problem with the language. It smells much more like a problem with me!\nThe bad(?) I (wrongly) set out with the intention of building an emulator which didn\u0026rsquo;t break out of the core rust safety model using runtime checks or unsafe blocks. As a consequence I spent a very long time trying to work out how to model an emulator in rust and I\u0026rsquo;ve ended up writing something in which data flow is far harder to reason about than I would have if I\u0026rsquo;d been allowed multiple mutable references \u0026amp; circular references.\nConsider the following diagram showing some elements of the NES\nHere the CPU needs mutable access to the cartridge (cartidge hosts RAM) and the PPU also needs mutable access to the cartridge. The CPU also needs mutable access to the PPU (set PPU data like palette information) but the PPU might trigger an NMI (non-maskable interrupt) to tell the CPU something. In my gameboy emulator (C#) I can model this domain exactly as above, everything has mutable references to everything else (roughly) and whilst that is less robust to future developers making mistakes it\u0026rsquo;s a closer model of the actual data domain in which I\u0026rsquo;m working.\nIn (safe, compile time checked) Rust the above has two key problems forcing me to create something a bit weird, less performant and harder to reason about:\nThe cartridge is owned by two objects (CPU \u0026amp; PPU both read/write to the cartridge) This was almost fun, cartridges have multiple components so I split the cartridge into a Box\u0026lt;dyn PpuAddressBus\u0026gt; and a Box\u0026lt;dyn CpuAddressBus\u0026gt;. Unfortunately there\u0026rsquo;s still some address lines which can mutate state on both. So very sadly I end up duplicating all writes from the CPU into both cartridge spaces (this isn\u0026rsquo;t as big a perf deal as you\u0026rsquo;d think, 99.9% of the time the write is ignored on one or other cartridge) There are cycles between the CPU and components which fire interrupts (PPU, APU \u0026amp; Cartridge) This was really ugly, I end up having to poll for interrupts from the CPU to each of the components, that in itself has probably caused literal days of pain handling subtle timing issues of when the poll occurs in relation to the write which caused the interrupt. All because I can\u0026rsquo;t have a reference cycle. The end result looks something like this:\nWhat\u0026rsquo;s my conclusion here? Is rust bad for my particular problem domain? Kind of yes and kind of no. A more experienced rust developer would have taken one look at the domain and said \u0026ldquo;yep, I need to use Rc\u0026lt;RefCell\u0026lt;\u0026gt;\u0026gt;\u0026rdquo; and not even tried to do what I did. I didn\u0026rsquo;t feel qualified to make that call at the start and crucially it would be a collosal PITA to refactor the whole domain at a late point in the project. In another language you would have expected to model the domain exactly and if you got it wrong you\u0026rsquo;d have been in a much better place to reorganise your code.\nI think what that really drove home for me is that architecting code in rust is notably harder than architecting it in another language, your choices are more limited, you will quite easily dig yourself a hole you can\u0026rsquo;t easily get out of (I nearly threw the whole thing away when I realised that Cartridges could fire IRQs (they only can on certain \u0026ldquo;mappers\u0026rdquo; so it was quite late into the project that I noticed)). Maybe this is controversial?\nDevelopment Environment Tooling The overall tooling for rust is obviously absolutely incredible, I don\u0026rsquo;t care which language you come from there\u0026rsquo;s nothing out there that can beat it. rustfmt, clippy, cargo bench, workspaces, cargo flamegraph, dependabot support etc etc. With the notable exception of code coverage there was really nothing that I felt like I was missing and crucially none of it was utterly bollocks unlike certain other languages. The documentation on how to use various features was excellent as well.\nGood job everyone involved in making rust not annoying to work with!\nIDE/Debugger I exclusively work in windows and have a very strong preference for a development environment with interactive debugger, integrated test results, one click to run various different invocations of the program etc. I started off with vscode but found the code completion and debugging support in Windows to be atrocious (failed to start sometimes, hit errors other times, breakpoints didn\u0026rsquo;t work intermittently, ability to inspect state on a breakpoint was flaky) so switched to CLion (Jetbrains C++ IDE with rust support). Fortunately that was excellent, paired with the visual studio toolchain and bundled LLDB I got something quite similar to the IDE experience I\u0026rsquo;m used to writing python/C#/java. Clippy warnings and proposed fixes inline, rustfmt on save, automatic macro expansion, step through debugging with variable inspection.\nI did have some issues after migrating the application into workspaces, occasional hangs whilst running background analysis processes and some fun guessing how to tell CLion what the invocation is to debug a binary but nothing insurmountable.\nProfiling Another thing I thought I would miss from visual studio c# coding was profiling tools, the profiling tools for dotnet are absolutely first class and I was expecting to run into the \u0026ldquo;oh, just use this arcane C thing but make sure you have the right symbols loaded and you\u0026rsquo;re using this exact version of blah\u0026rdquo;. Fortunately not true (smile). I was unable to cargo flamegraph due to dtrace requirements (c.f. https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902 - it\u0026rsquo;s coming but only insider builds of windows at the moment). However, because I was building for windows with the toolchain stable-x86_64-pc-windows-msvc I was able to attached visual studio to the process and get a really nice output like I\u0026rsquo;d expect.\nEnough to fix my stupid \u0026ldquo;oh yeah, constructing strings for logging is expensive I remember\u0026rdquo; issue at any rate (wink)\nCI/CD Emulators are incredibly finicky beasts, tweaking things to fix one issue will often break several tests that relied on precise timings elsewhere. So automated testing is basically a necessity. I wasn\u0026rsquo;t sure what sort of support I\u0026rsquo;d get in Azure DevOps for rust but I was at least able to get it building on nightly, mac, windows and linux on each commit along with parsing test results. I normally like to get code coverage stats as well but it seemed less trivial than I\u0026rsquo;d hoped and I haven\u0026rsquo;t yet got around to it. Definitely not insurmountable and I\u0026rsquo;ve seen posts floating that suggest this\u0026rsquo;ll be trivial in the future.\nI found https://github.com/crate-ci/azure-pipelines after writing my pipeline based on blog articles and it looks to be a much more comprehensive set of ideas. So nobody should be looking at what I did for ideas, just noting that a beginner in both ADO and rust was able to get up and running with minimal fuss!\nConclusion \u0026ldquo;Rust restricts how you model your domain, and I\u0026rsquo;m not sure that\u0026rsquo;s necessarily a good thing\u0026rdquo;\nA good principle to remember when considering coding arguments is \u0026ldquo;my problems are not necessarily your problems\u0026rdquo;. I think all senior engineers understand this at some point. When you find yourself inclined to argue about coding issues: \u0026ldquo;unit tests vs integration tests\u0026rdquo;, \u0026ldquo;dynamic vs static typing\u0026rdquo;, \u0026ldquo;short functions vs long function\u0026rdquo; or any other absurd thing that you can bait people with then I think this is a great mantra to remember. Not all domains are equal, someone working on websites has a very different outlook to someone writing FPGA code to someone writing games. Typically similar problems exist in all these environments but the value functions applied to determine how and whether to solve them are different.\n8 bit Emulators as a domain are quite interesting, they require no(ish) dynamically allocated memory, have very stringent performance requirements whilst being incredibly poorly suited to concurrency and have an exceptionally well defined end point. So taking as an example, Rust cares very deeply about data ownership which helps make sure that concurrent code is safe. Great! \u0026ldquo;Your problems are not my problems\u0026rdquo;.\nSo most of rusts restrictiveness of structure is unnecessary in an emulator environment. Therefore, when it encourages a poor domain model it feels like a bad trade off. Now, that said, it\u0026rsquo;s clear that the correct domain model is viable if you know how to make proper use of https://doc.rust-lang.org/1.15.1/book/choosing-your-guarantees.html. So my real conclusion is that rust forces an up front decision which only experienced rust developers who fully understand the domain they\u0026rsquo;re about to model can get right.\n","permalink":"https://blog.davetcode.co.uk/post/nes-emulator-rust/","summary":"Writing a NES Emulator in Rust with no unsafe code and only compile time guarantees. Does rust provide good value when writing an emulator?","title":"NES Emulator in Rust"}]