Address marking revisited

A basic requirement for multiprocessing with Coops is a
mechanism to map object identifiers to their processors
globally.  

In Coops 1.x, objects are identified by the address of their
instance variables in the heap.  Node numbers are introduced
to identify non-local objects.  The node number is stored in the
upper unused bits of the object address.

This method has proven unsuitable for many processors and 
operating systems because the upper address bits are allocated
and used by the physical memory architecture.

Coops 4 was introduced to overcome this difficulty by using
an object table.  Object identifiers are integers that index the
object table.  The table contains the local address of the object.
Non-local objects are again identified by a node number which can
be stored in the upper bits of the object index.

The object table solution has another advantage: In Coops 1.x,
the 'objects' data type is declared as a union so objects can
be cast into appropriate types.  Most C compilers treat union
struct types less efficiently as return values or parameters to
functions. Coops 4 objects are simply integers, so the compilers
make better code for manipulating them.

Against this advantage is the overhead of accessing the
instance variable pointers through the object table. (Done with
the 'begin' macro in every method.)  In addition, the object table
itself is a large fixed-size array that must be allocated for worst
case conditions on each processor.

A better way?

The best solution would be a global address assignment strategy so
that both objects and buffer pointers could be identified with their
processors and addresses could be used directly without an object
table.  This would only be possible if every processor had a unique
address space so that all object addresses would be system-wide unique.

Apparently, this result could only be realized by building our own
processor boards for both host and DSP processors; however there is
a way to achieve the same result with all processor and host
architectures currently using Coops.  Further, this change is minor in
scope and does not require significant modification to the Coops 1.x
source code.

The basic idea is to mark objects with the low order address bits 
rather than the high bits.  This method will waste a small amount
of memory, but it overcomes very large difficulties.

To allow up to eight processors, we use the low order three bits of
every object address.  Because DSP's allow odd addresses, we allocate
all DSP objects so they have the low order address bit set.  This
leaves two more bits to identify the processor:

	001	DSP 0
	011	DSP 1
	101	DSP 2
	111	DSP 3

For host objects we normally have only even address, so for these
objects the low order bit is always zero and we have four more
processors:

	000	HOST 0
	010	HOST 1
	100	HOST 2
	110	HOST 3

The hosts, of course, may be distinct applications or DLL's on a single
host workstation.

When heaps are configured, they are given a node identifier for their
low order bit pattern.

When an object is allocated from a heap, the addresss is incremented
until the proper low order bit pattern is set.  This wastes a small
amount of memory: The average loss will be four bytes on a byte address-
able processor and four long words on a DSP.

Because objects are identifed by their actual address, no bit
manipulations are needed to access the instance variables locally.
Further, non-object allocations of buffers/strings/arrays etc.
can be identified with their processors as well.





Each of the address bits in the low order of pointers can be grouped
to provide a node number.  The loss is that each processor's heap
module must arrange to allocate objects on the appropriate boundaries.

The DSP's have sequential long word addresses, so they alone can have
odd addresses.  

000
010
110
100

000
000

The three low order bits of an address will waste eight bytes on
byte addressed processors and eight long words on DSP processors
per object.

Because the low order bit on byte addressed processors generally must
be zero, we can identify four distinct processors by using the other
two bits.

For DSP processors, we make the low order bit ALWAYS equal 1,
The other two bits can then identify four disting DSP processors.

The cost is in the heap management: Up to eight long words may be wasted
per object on the DSP and eight bytes on the other processors.  This 
number is statistical however, and a reasonable estimate of the average
loss is half this figure, 4 bytes or 4 long words respectively.

The effect of using unions can also be overcome by 