Exquisite Python hack — Smalltalk-style class extensions

February 10, 2009

I’ve discovered (or rediscovered) an exquisite hack — a way to add or (in this case) replace methods of an existing but out-of-scope module, and then put them back again.

Motivation

The minimal ZeeUnit is intended to discover the minimal Zeetix kernel. It therefore should not depend on any clazzes that are outside the Kernel assembly.

In order to reliably restore the underlying database (from which the Kernel clazzes are loaded), it uses code within the existing kernel that (incorrectly) requires clazzes from two non-kernel assemblies — Admin and Command.

I needed a way to use the existing kernel code, while not adding these two big warts to ZeeUnit. I therefore cloned the required clazzes into KernelTest.CoreTest, and made minor edits so that they work.

Unfortunately, two kernel clazzes — ItClazz and ItRecordClazz — contain methods that expect to find these missing clazzes in their original location.

I therefore needed a way to temporarily replace the offending methods so that the new test code works properly.

Solution

I borrowed a paradigm from Smalltalk — Smalltalk-style class extensions. A class extension, in Smalltalk, is a group of methods that can be atomically added or removed from an existing Smalltalk class. This behavior is simulated in Ruby by the Ruby “module” mechanism.

It turns out that a Python class uses a dictionary-style mechanism, similar to Smalltalk, for method dispatch. Python supports adding and removing methods from classes, at run time, by assignment operations on the class. The trickiest part is determining the specific Python class that is associated with a given Zeetix clazz (or Metaclazz).

I used the setUp and tearDown hooks in TestCase to install and then remove the special methods, leaving the Kernel unaffected except while running the test(s) in question.

Result

The result is that the special behavior added by the new test case is available to every existing instance of every Zeetix object, without modification. This proved extraordinarily valuable in this case, because the desired behavior is used across the clazz and metaclazz hierarchy. The resulting code works like a charm.
Read the rest of this entry »

Advertisements

Repairing insidious database bug

February 10, 2009

I’ve added a slew of new tests that validate the persistence clazzes (ItRecord descendants).

They exposed an insidious bug, that I’m now repairing: SetRecord and SortedCollectionRecord have never been correctly loaded. Worse, the record_id of their behavior entry is wrong — it is “0”, which should never happen.

This, in turn, means that I now have to adjust the database cleanup and setup routines so that they can be called from regular code, instead of the browser-based admin tools. I suppose I’ll have to be disciplined and write a new test case that confirms THIS change.

The database cleanup and setup routines are invoked from “prepareForDebug”, which is located someplace in the ZeeLife root. I suppose now is as good a time as any to go find and capture it.

In addition, all this means that I’ve started to populate the sql branch of the git tree. I need to move the Kernel.ZeeStore records around. The sql branch is organized like the python and ruby branches; the sql for each assembly/subassembly is located in a correspondingly-named subdirectory of the sql branch.

Create a new clazz

February 9, 2009

For a class named “Foo” in assembly “Bar”:

  1. Create a sql file that adds suitable records to it, behavior, clazz, and metaclazz, and any superclazzes. If Foo is to have persistent instances, this file should also create a new table named “foo”. By convention, the sql file is named “create_foo.sql”, and is kept in the “sql” subdirectory of the “Bar” directory. NOTE: Each new object requires a new version_id/object_id, and these must be kept in synch with the ZeeStore.
  2. Run, from either the command line or MySqlQueryBrowser, the sql created in the previous step. Validate that the new records exist in the ZeeStore and are well-formed.
  3. Create a python file containing class definitions for the new clazz and its metaclazz. The new file should be named “Foo.py”, and should be located in the “Bar” subdirectory of the zeetix root. Inside “Foo.py”, there should be an import statement that imports the superclazz, and class definitions for “Foo” and “FooClazz”.
  4. Create another python file containing class definitions for the record clazzes of the new clazz. The new file should be named “FooRecord.py”, and should be located in the “Bar” subdirectory of the zeetix root (alongside “Foo.py”). Inside “FooRecord.py”, there should be an import statement that imports the superclazz (another descendant of ItRecord), and class definitions for “FooRecord” and “FooRecordClazz”.
  5. Add implementations of hook methods required by the various framework interfaces, such as “doTransientAttributeNames”, “doTableName”, and so on.
  6. Add an import line and installation code to “BarAssembly.py” that imports and installs Foo and FooRecord.

With proper tweaking of the existing Zeetix naming conventions, it should be possible to generate the required column names (in the DB), record attribute names (in the record clazz), and instance attributes (in the new clazz) from the specification provided in the original subclazz creation method. I do some of this now (I use ZeeString), and could do MUCH MUCH more.

Add persistence support to the kernel

February 9, 2009

We need to incorporate persistence into the tests that document and validate the kernel. Each clazz that can have persistent instances must have a corresponding record clazz. The kernel depends on this behavior (each kernel clazz is loaded from the ZeeStore), and the resulting functionality is a primary aspect of the very reason for Zeetix to exist.

I think (although I’m not sure) that every clazz, even those that do not have persistent instances, should have a corresponding record clazz that describes the clazz itself. There is a potential loop here, which I remember thrashing at for extended periods, about the relationship between instances of MetaclazzRecord, ClazzRecord, and so on.

I have since gotten into a bad habit of not adding transient clazzes (those that I know will never have persistent instances) into the ZeeStore. This will break as soon as we attempt to automatically generate non-python source code for those clazzes, because the ZeeStore will not contain the records needed to construct them.

Meanwhile, I need to document the recipe for creating a new clazz, so that we can add behavior to automate it. My bad habit is a result of the tedium of editing and updating the SQL and record clazzes when I didn’t absolutely need it.

The new tests should validate at least the following existing clazzes:

Kernel.Core.BehaviorRecord
Kernel.Core.BlockRecord
Kernel.Core.ClazzRecord
Kernel.Core.ItRecord

Kernel.ClazzDevelopment.ClazzCreatorRecord
Kernel.ClazzDevelopment.ClazzDefinitionDescriptorRecord

Kernel.CLDT.AdditiveSequenceableCollectionRecord
Kernel.CLDT.ArrayedCollectionRecord
Kernel.CLDT.ArrayJoinRecord
Kernel.CLDT.ArrayRecord
Kernel.CLDT.CollectionRecord
Kernel.CLDT.DictionaryRecord
Kernel.CLDT.KeyedCollectionRecord
Kernel.CLDT.OrderedCollectionRecord
Kernel.CLDT.ReferenceRecord
Kernel.CLDT.SequenceableCollectionRecord
Kernel.CLDT.SetRecord
Kernel.CLDT.SortedCollectionRecord
Kernel.CLDT.SystemDictionaryRecord
Kernel.CLDT.ZeeManyToManyStitchRecord
Kernel.CLDT.ZeeManyToOneStitchRecord
Kernel.CLDT.ZeeOneToManyStitchRecord
Kernel.CLDT.ZeeOneToOneStitchRecord
Kernel.CLDT.ZeePairJoinRecord
Kernel.CLDT.ZeeStitchRecord

Kernel.ZeeStore.ZeeMemoRecord

Perhaps these new tests should be incorporated into the tests for each  Kernel (sub)assembly.

TestSuiteTestCase added to ZeeTestTest

February 8, 2009

The new ZeeSuiteTestCase demonstrates the aboutToRun and finishedRun bugs that I found in the original SUnit, and therefore ZeeUnit, test suite.

Interestingly, it demonstrates the utility of clazz instance variables. In this case, a message (aboutToRun and finishedRun) was being sent to all descendants of TestCase instead of just the TestCase descendants whose instances were contained in  whatever TestSuite instance was being run.

I created a descendant — CountingTestCase — that contains two clazz instance variables (“ranAboutToRun” and “ranFinishedRun”). It hooks the aboutToRun and finishedRun clazz methods and, within each, sets the corresponding clazz instance variable before delegating to its superclazz (TestCase). It also contains a trivial instance method, “dummy”, that TestSuiteTestCase uses to instantiate test instances. I then created two descendants of CountingTestCase — “ShouldNotRunTestCase” and “ShouldRunTestCase”.

The tests within TestSuiteTestCase (there are currently four of them, two each for aboutToRun and finishedRun) create a new TestSuite and add an instance of ShouldRunTestCase that invokes the “dummy” test method. It then asserts that the ShouldRunTestCase clazz method was run and denies that the ShouldNotRunTestCase clazz method was run.

The initial version of ZeeTest (and SUnit) fails this test.

I then modified the aboutToRun and finishedRun instance methods of TestSuite to fix the problem (see the “Python closures and collections are broken” flame for more).

I then committed and pushed the new code to the git repository.

Now that this ZeeTest bug is fixed, I can resume working on the KernelTest.CLDTTest and KernelTest.AssemblySupportTest subassemblies.

Python closures and collections are broken

February 6, 2009

It is too bad that “lambda”, in Python, is so misleadingly named. It is even more regrettable that the scoping semantics of Python’s attempted closure mechanism are equally broken. Python’s “lambda” is a very thin syntactic veneer on top of Python’s local function behavior.

The specific problem is that the scope of a local function does not nest inside its containing scope(s). Here is an example:
Read the rest of this entry »

TestSuite>>finishedRun is broken

February 6, 2009

Problem Description

The finishedRun message follows the same pattern as TestSuite>>aboutToRun, is broken in the same way, and requires the same fix.

This bug also demonstrates that a test case is missing from ZeeTestTest (and, for that matter, from the original Smalltalk unit test test). The test case needs to be created and the bug repaired.

At a meta-level, it seems clear enough that these two bugs are really ONE bug with two symptoms. This, in turn, suggests additional required functionality in the Zeetix environment. It should be possible to express the relationship between these “two” bugs, so that it can be fixed “once and only once”.

At a deeper level, TestSuite>>aboutToRun and TestSuite>>finishedRun are two applications of the same underlying code pattern. That pattern originally delegated each to the clazz. Instead, it should have delegated each to a (missing) enumeration of the contents of the receiver.

This deeper relationship between these two apparently (from the source code) unrelated methods cannot be expressed in a notation-driven environment. This inability demonstrates why the Zeetix approach is needed.

Problem Code

The broken code is the implementation of the instance method (finishedRun) of TestSuite. The current Python implementation (transliterated from Smalltalk) is:

clazz TestSuite:
    #...
    def finishedRun(self):
        self.clazz().finishedRun()

This should be replaced with an implementation that enumerates all tests contained in the receiver and sends finishedRun to each. In Smalltalk, this would read something like the following:

TestSuite>>finishedRun
   self allTestCases do: [each | each finishedRun]

The allTestCases method does not currently exist and needs to be created.
To Do

  1. Create a test case in ZeeTestTest that demonstrates the bug.
  2. Modify the code in ZeeTest to fix the bug
  3. Demonstrate that ZeeTestTest passes all tests, including the new one.

TestSuite>>aboutToRun is broken

February 6, 2009

Problem Description

The aboutToRun message is supposed to be sent to each testCase before it runs. The implementation, inherited from SUnit, instead sends aboutToRun to each descendant of TestCase. This is particularly problematic when a descendant of TestSuite is created and instantiated — it initiates a circular reference and fails.

This bug demonstrates that a test case is missing from ZeeTestTest (and, for that matter, from the original Smalltalk unit test test). The test case needs to be created and the bug repaired.

Problem Code

The broken code is the implementation of the instance method (aboutToRun) of TestSuite. The current Python implementation (transliterated from Smalltalk) is:

clazz TestSuite:
    #...
    def aboutToRun(self):
        self.clazz().aboutToRun()

This should be replaced with an implementation that enumerates all tests contained in the receiver and sends aboutToRun to each. In Smalltalk, this would read something like the following:

TestSuite>>aboutToRun
   self allTestCases do: [each | each aboutToRun]

The allTestCases method does not currently exist and needs to be created.
To Do

  1. Create a test case in ZeeTestTest that demonstrates the bug.
  2. Modify the code in ZeeTest to fix the bug
  3. Demonstrate that ZeeTestTest passes all tests, including the new one.

Determine how to test for "Bug Patterns"

February 4, 2009

Certain kinds of bugs tend to be pervasive throughout the framework. In Python, for example, methods must explicitly return a value. How do I write a test that finds and fixes those that don’t?

Example: Each Clazz that has transient attributes should implement the “doTransientAttributeNames” hook. A typical example should look like this:

def doTransientAttributeNames(self):
    answer = [
        'attributeOne',
        'attributeTwo',
        ]
    return answer

I often forget the final line, and this in turn causes a recognizable symptom in the debugger — a “NoneType” exception in It>>allTransientAttributeNames.

How do I write a test that traverses implementors of doTransientAttributeNames and fails for those in which the “return” is missing?

How do I supply a list of hooks that require a return value (not all do) and apply the above test for each hook (in each clazz)?

How do I automatically incorporate this test into the interface specification of a clazz, so that when a new clazz is created this test is automatically applied to it?

Adding a new Assembly/SubAssembly

February 4, 2009
  1. Create the assembly directory.
  2. Add the __init__.py module to the assembly directory.
  3. Create the assembly module in the assembly directory.
  4. Add the clazzes in the assembly to the assembly module.
  5. Create a clazz module for each clazz in the assembly directory.
  6. Add an import for the new assembly to SystemConfiguration.py (ouch!)
  7. Add the assembly to the doAssemblyNameList method of each application that requires the new assembly.