Archive for the ‘Minimal Kernel’ Category

Exquisite Python hack — Smalltalk-style class extensions

February 10, 2009

I’ve discovered (or rediscovered) an exquisite hack — a way to add or (in this case) replace methods of an existing but out-of-scope module, and then put them back again.

Motivation

The minimal ZeeUnit is intended to discover the minimal Zeetix kernel. It therefore should not depend on any clazzes that are outside the Kernel assembly.

In order to reliably restore the underlying database (from which the Kernel clazzes are loaded), it uses code within the existing kernel that (incorrectly) requires clazzes from two non-kernel assemblies — Admin and Command.

I needed a way to use the existing kernel code, while not adding these two big warts to ZeeUnit. I therefore cloned the required clazzes into KernelTest.CoreTest, and made minor edits so that they work.

Unfortunately, two kernel clazzes — ItClazz and ItRecordClazz — contain methods that expect to find these missing clazzes in their original location.

I therefore needed a way to temporarily replace the offending methods so that the new test code works properly.

Solution

I borrowed a paradigm from Smalltalk — Smalltalk-style class extensions. A class extension, in Smalltalk, is a group of methods that can be atomically added or removed from an existing Smalltalk class. This behavior is simulated in Ruby by the Ruby “module” mechanism.

It turns out that a Python class uses a dictionary-style mechanism, similar to Smalltalk, for method dispatch. Python supports adding and removing methods from classes, at run time, by assignment operations on the class. The trickiest part is determining the specific Python class that is associated with a given Zeetix clazz (or Metaclazz).

I used the setUp and tearDown hooks in TestCase to install and then remove the special methods, leaving the Kernel unaffected except while running the test(s) in question.

Result

The result is that the special behavior added by the new test case is available to every existing instance of every Zeetix object, without modification. This proved extraordinarily valuable in this case, because the desired behavior is used across the clazz and metaclazz hierarchy. The resulting code works like a charm.
(more…)

Advertisements

Repairing insidious database bug

February 10, 2009

I’ve added a slew of new tests that validate the persistence clazzes (ItRecord descendants).

They exposed an insidious bug, that I’m now repairing: SetRecord and SortedCollectionRecord have never been correctly loaded. Worse, the record_id of their behavior entry is wrong — it is “0”, which should never happen.

This, in turn, means that I now have to adjust the database cleanup and setup routines so that they can be called from regular code, instead of the browser-based admin tools. I suppose I’ll have to be disciplined and write a new test case that confirms THIS change.

The database cleanup and setup routines are invoked from “prepareForDebug”, which is located someplace in the ZeeLife root. I suppose now is as good a time as any to go find and capture it.

In addition, all this means that I’ve started to populate the sql branch of the git tree. I need to move the Kernel.ZeeStore records around. The sql branch is organized like the python and ruby branches; the sql for each assembly/subassembly is located in a correspondingly-named subdirectory of the sql branch.

Day four

February 4, 2009

ClazzDevelopmentTest

I’ve defined a new test subassembly, ClazzDevelopmentTest, with three new test cases:

  • ClazzDevelopmentLoadsTest: Validates that the clazzes that comprise the ClazzDevelopment subassembly are present and well-formed.
  • BuildZeetixKernelTest: Validates that the clazzes that comprise a newly-formed instance of SystemConfiguration are properly built. The tests in AssemblySupportTest validate new SystemConfiguration instances — the tests in BuildZeetixKernelTest validate the machinery that builds them.
  • BuildNewSubclazzTest: Validates that the environment can create properly-formed new subclazzes at runtime and install them into the running ecology. This machinery is used throughout the rest of the universe, and so its proper operation is crucial.

Refactored KernelTest hierarchy

Most of the tests in the KernelTest subassembly hierarchy need to validate Kernel clazzes for correctness. Since every Clazz is built to the same pattern, this validation code is common to all of them. Rather than copy the validation code among each TestCase descendant, I refactored the module hierarchy so that those tests inherit from an abstract “ClazzValidator” (itself a descendant of TestCase) that provides the validation behavior.

I’m aware that this might better be accomplished by applying the strategy pattern, relying on delegation instead of inheritance, but this works for now. It is “the simplest thing that can possibly work.”

This has two immediate benefits:

  • The validation code is collected in one place (Once and Only Once).
  • The SystemConfiguration clazz, created specially, is validated using the same test as all other kernel clazzes. This ensures that subsequent changes to clazz structure are correctly propagated to all clazzes

A “landmine” bug found and fixed

Part of the SystemConfiguration validation test is to verify that the size of the global namespace (“them”) is non-zero. As it turns out, the size reported by SystemDictionary (the clazz of “them”) is always zero. Ouch. The dictionary exists and is the correct size; it is the method that answers the size (“doSize”) for just this Collection descendant (SystemDictionary) that is broken — Set and SortedCollection work just fine.

This, again, is the sort of bug that is pernicious and hellishly difficult to track down. The instance works most of the time. Debuggers report its size correctly. When the bug causes a problem, the symptom is often far removed from the source — in both time and lexical space. The code that needs the size may likely be correct. The code that contains the bug ran a long time ago (at startup, in this case) and in a faraway place (in the Kernel.Core.CLDT subassembly).

I call a bug like this a “landmine” bug (not to be confused with another equally pernicious category, the “timebomb” bug). A landmine bug is a bug that makes a system work nearly all the time, until you step on the landmine. Then — “boom” — the landmine goes off, and the developer is left wondering “what happened?”

This means that the KernelTest.CLDTTest subassembly needs to include a test case that validates that each concrete descendant overrides doSize to answer the correct value (for that descendant, of course).

The more general learning is that we will need a way to specify, as part of an interface, that descendants of a clazz correctly implement the hooks required by that clazz. This, in turn, means that the universe needs a family of interface abstractions and an associated way to test them (see below).

I love unit tests.

KernelTest.CLDTTest subassembly

Validates the existence and correct behavior of the Kernel.CLDT (Common Language Data Types) clazzes. In this case, these are the clazzes that support collections with the Zeetix universe.

to do: Create a test case that validates that doSize is correctly implemented in each concrete Collection descendant.

InterfaceSupportTest and InterfaceSupport

Validates that the InterfaceSupport clazzes exist and operate correctly. When a resource, assembly, or feature claims compatibility with a specific interface, these tests validate that claim. When a resource, assembly, or feature claims to provide a specific existing existing or new interface, these tests validate that claim.

Day three

February 3, 2009

I committed and pushed yesterday’s new stuff to the github repository. I’m blown away by the power and functionality of git/github.

Plan

  1. Clone the new Python test cases (KernelCoreLoadsTest and SystemConfigurationTest) into the Ruby branch.
  2. Design and build test cases that validate the behavior of the Kernel.ClassDevelopment subassembly (ClazzCreator, ClassDefinitionDescriptor, ZtxWorldBuilder). This subassembly builds ALL the subclazzes used by Zeetix, after the initial Kernel is loaded — assuring that each subclazz is built the same way. This subassembly also provides the behavior that does the right thing for “schema migration” — changes to a clazz with both instances and subclazzes. This is particularly crucial for Zeetix because a Zeetix ecology is always alive — it must be possible to create, load, unload, and forget clazzes on the fly.
  3. Design and build test cases that specify Assembly/SubAssembly behavior. This distinction is missing from the spike solution, and needs to be there. A subassembly is loaded and unloaded atomically, and inherits its prerequisites (and therefore dependencies) from the assembly that contains it. An Assembly determines which subassemblies to load based on runtime constraints such as language choices, platform capabilities, and so on.
    Design and build test cases that exercise loading persistent clazzes from the zeestore. While this behavior doesn’t seem critical to get ZeeUnit up and running — and therefore perhaps should be deferred — it seems so central to the larger paradigm that I fear the consequences of ignoring it. I know, I know — “YAGNI”. I’ll try to resist.

Day 2 progress

February 2, 2009

KernelCoreLoadsTest done

KernelCoreLoadsTest is running in Python. It demonstrates that SystemConfiguration, It, Behavior, Clazz, and Metaclazz are each present, along with their metaclazzes.

SystemConfigurationTest

SystemConfigurationTest is running in Python. This demonstrates that SystemConfiguration and its metaclazz are correctly wired together and correctly installed into the kernel core framework. This clazz is built “by hand”, instead of by the Kernel.ClazzDevelopment assembly, because portions of it are needed while the Kernel metastructure is being built.

This test exercises the behavior that was broken when I began this exercise, so that it doesn’t get broken again in the future.

First bug found and fixed!

February 2, 2009

One of the more important pieces of Zeetix plumbing is SystemConfiguration.

This clazz loads the current “ecology” (something like a Smalltalk image or Lisp environment). It’s loaded with clazz methods that do important things like load the kernel clazzes, construct the kernel namespace, and so on.

The very first test is “KernelClassesLoadedTest”, verifying that all is present and correctly stitched together. The very first clazz tested is SystemConfiguration. Boom. The clazz exists (nothing would work otherwise), but is missing from the global namespace. Bad bad bad. Zeetix MUST have the ability to load and unload assemblies (collections of clazzes, something like Ruby Gems), and that in turn demands run-time access to SystemConfiguration — the clazz, not the instance.

This would have been very difficult to track down, because the current instance is available from any object, and that instance points to its clazz. So it seems to work — until you try to, for example, get a new ecology (from the clazz, using the global namespace). It’s BEEN missing all along. Not any more.

Chalk up another big win for test-first development. Why didn’t I start this a long time ago?

Getting started

January 31, 2009

Goal

Build the minimum needed to make a working ZeeTest binding for python, ruby, and javascript.

Context

A spike solution (with minimal tests) with server-side code in Python and browser code in Javascript works. The existing kernel code is bloated and contains naive and bogus code that I wrote years ago when I understood neither Python nor Zeetix. Buried inside that is the prize.

Strategy

  1. clone the relevant parts from Python into a Python binding of ZeeForge, driven by the requirement to make ZeeTest (and ZeeTestTest, ZeeTestUI::TestApplication, and so on) work.
  2. Construct and rely on test case assemblies in KernelTest — CoreTest, AssemblySupportTest, and others — to assure the continued correct behavior of the relevant Kernel modules as I remove the crud.
  3. When done (or along the way?), I clone/copy the result to Ruby, ensure that the Ruby binding also works.

Meanwhile, I need the minimal SQL and database that supports it. I think that’s there, I wonder how this is tested. This sounds like a TestResource (for other tests), and also has its own tests.

New Tests

  • KernelTest.KernelCoreTest.KernelCoreLoadsTest: Validates that the Kernel.Core clazzes have been loaded.
  • KernelTest.AssemblySupportTest.AssemblyLoadUnloadTest: Validates that assemblies can be loaded and unloaded.

Notes

  • I created KernelTest to hold the tests needed to preserve the integrity of the Kernel.
  • Created KernelCoreLoadsTest in KernelTest, because the CommandLineLauncher needs to have the KernelCore in order to do anything uself.
  • This already requires me to add KernelTest and such to SystemConfiguration (in Kernel/AssemblySupport), currently as an explicit import. This is WRONG WRONG WRONG. I need to load and unload assemblies — so I need to add tests to KernelTest/AssemblySupportTest
  • Thus, I created “AssemblySupportTest/AssemblyLoadUnloadTest”.
  • AssemblyLoadUnloadTest will have profound implications for SystemConfiguration, because assemblies will need prerequisite/dependent machinery — machinery that doesn’t exist today.
  • As I discover needed tests in Ruby, I’m building them in Python and validating that they work. The plan is to then use those tests to keep things working while I slash unnecessary junk from the existing Python code.