This is the second article about Jumbo.
As I said previously, Jumbo originally ran on Mono and the .Net Framework. Part of my hesitation in releasing this project was because of how complicated it could be to get it running on that environment.
When .Net Core started getting more mature with the release of .Net Core 3, I was interested in learning more about it. While I don't use .Net in my day-to-day anymore, I still like it, and .Net Core's promise of cross-platform .Net would've saved me so much hassle back in the day. So, I was curious to see whether I could get Jumbo running on it, just for fun.
Porting was, fortunately, not that difficult. To make things easier on myself, I started in a clean workspace, and copied over Jumbo's components one at a time, created SDK-style projects for them, and fixing build errors until they compiled. Some changes were necessary, but it wasn't too bad.
One early difference I ran into was how sockets worked. On Linux, if you listen on an IPv6 address, you automatically also listen on the corresponding IPv4 address. If you subsequently also tried to bind to the IPv4 address, it would fail. Jumbo handled this by only listening on IPv6 on Linux, but on both IPv6 and IPv4 on Windows. This worked great for Mono.
But, I guess .Net Core handled this differently, since it no longer listens on IPv4 automatically on Linux, so I had to change how Jumbo behaved. Fortunately, that only meant changing the default for the associated setting in Jumbo's configuration files.
One of the biggest issues I ran into was the lack of the AssemblyBuilder.Save method. This method,
needed to save dynamically generated assemblies to disk (which Jumbo's JobBuilder depends on),
doesn't exist in .Net Core (and still doesn't in .Net 6). Fortunately, I was able to find a NuGet
package that provided the same functionality, Lokad.ILPack,
which saved the day. Unfortunately, it had a bug
that prevented it from working for my scenario, but I was able to find and contribute a fix.
Another minor issue was the lack of AppDomains. Jumbo's TaskServer normally runs tasks in an external process, but it could run tasks in an AppDomain if requested, which was used to ease debugging and testing. Since AppDomains don't exist anymore, I had to cut this feature. But, since it isn't used when running Jumbo normally, it wasn't a big loss. The biggest difference is that it does make running the unit tests a bit more fragile, since they now depend on the external TaskHost process.
Then, I discovered that .Net Core can't serialize delegates, but I was able to work around that too. With that, I finally got all the tests working. Not only that; I could run them on Linux now, which was never possible before (old Jumbo's tests depended on NUnit and only ran on Windows).
Another interesting part was the CRC library. Jumbo's DFS calculates checksums a lot, so original Jumbo used a native library, for a faster implementation than the managed version. That library was built as part of Jumbo, either with Visual C++ (on Windows), or g++ (on Linux). Integrating that native build into the .Net Core build process turned out to be a hassle.
I did some speed tests to find out if this would still matter, and tried to optimize the managed fallback algorithm a bit more. And then, I found a CRC32 library for .Net Core that was even faster than my original native implementation, so that solved that problem.
Oh, and Ookii.CommandLine 2.3's inclusion of a .Net Standard 2.0 binary? That's because I needed it for this Jumbo port! Otherwise, I'm not sure when I would've gotten around to that.
Finally, I just had to remove some of the Mono-specific parts from the code (there are probably still some leftovers), and it was good to go. Except...
The least straight-forward part of the port was, without a doubt, the two administration websites, DfsWeb and JetWeb. These were written using ASP.NET Web Forms (with code-behind files and everything), which just wasn't supported in .Net Core. Not wanting to give up now, I persevered, and rewrote both projects to use ASP.NET Core, replacing the Web Forms with Razor pages. Fortunately, neither site was particularly big, so there were only a few pages to convert.
This also had a big advantage: to use the websites, previously you needed IIS Express (on Windows) or Mono XSP. Now, they could just run using Kestrel, the web server included with ASP.NET Core.
Jumbo was meant to run on a Linux cluster, and used a bunch of Bash scripts for deployment and running. These scripts used password-less SSH to start everything on all the nodes in the cluster, and were tailored only for the Linux environment. Running it on Windows, only used for testing purposes, was a mostly manual affair.
When I originally started planning to release Jumbo, back in 2013, I realized that maybe some people would want to be able to try out Jumbo, if only in a one-node configuration, on Windows, without having to manually launch multiple executables. I wrote some PowerShell scripts to facilitate that, but of course, those were for Windows only, so now there were two sets of scripts for basically the same thing.
With the port to .Net Core, there was something else interesting I could use: PowerShell Core! Now, I could adapt the PowerShell scripts to work on both Windows and Linux, and get rid of the old Bash scripts, rather than having to maintain two separate, functionally identical sets of scripts. PowerShell Core even supported remote sessions over SSH now, so could be used to launch Jumbo on multiple nodes in a Linux cluster.
Come to think of it, there's really nothing to stop you from running a hybrid cluster with both Windows and Linux nodes. I've never tried doing that. It could be interesting; it would probably be somewhat difficult to get the configuration right, since e.g. the scripts assume Jumbo is in the same location on every node, and the Windows paths need a drive letter. But it's probably possible to hack something together.
Anyway, that's the story of how Jumbo was brought into the modern age, and is no longer a pain to run on either Windows or Linux. I eventually brought it forward to .Net 6 as well (which was very straight-forward at that point), which is what's available now.
Check out both Jumbo's .Net Core port and the original .Net/Mono version on GitHub, and try to spot the differences!
No comments here...
Comments are closed for this post. Sorry.
Subscribe