Re-factoring

Prior to starting on the UI I have been doing some re-factoring. I have read that the important thing when re-factoring is that the code still works. My experience supports this. The ideal situation is that you re-factor in small steps and intersperse these steps with running unit tests that will ensure that you haven't broken anything. In the case of Cello as most of the work I have done is for functionality that allows Cello to mesh onto OSX, I have also written a lot of Unit Tests so I feel relatively secure about a modest amount of re-factoring.

The main things that I am doing re-factoring is:

• Working through consistent naming.
• Wrapping most the core foundation functions as methods.

Wrapping of core foundation functions as methods is something that I first came across in ACS (a part of MacApp). ACS aimed to present the what was originally the tool-box functionality as a set of C++ objects. In later versions following the introduction of CoreFoundation it did this, where possible, by defining the anonymous types. So in the case of CFData, a CFDataRef and CFMutableDataRef are defined as:

typedef const struct __CFData * CFDataRef;
typedef struct __CFData * CFMutableDataRef;

What ACS did was to then define the anonymous __CFData and add methods to it. So to add a method that would wrap CFDataGetLength it would did something like this:

struct __CFData
{
 CFIndex GetLength() const;
};

and then defined an implementation for GetLength in terms of CFDataGetLength

inline CFIndex __CFData::GetLength() const
{
 return CFDataGetLength(this);
}

When I started work on the Cello project I actively resisted this in favor of calling (typically) core foundation directly. What I found was that it made life a lot easier if I wrapped all the 'create' or 'copy' functions so that they returned boost::intrusive_ptr wrapped CF types. I did this as I used boost::intrusive_ptr to take care of all the ownership/ownercounting issues. When working on the Sound part of Cello I 'rediscovered' this ACS style wrapping, partly because it just made the code easier to read.

It means that instead of calling:

int length = CFDataGetLength(myData);

You write:

int length = myData->GetLength();

If you are dealing with boost::intrusive_ptr it eleminates a lot of 'get' calls so instead of

int length = CFDataGetLength(myData.get());

You write:

int length = myData->GetLength();

The other thing that comes into play is that I have had to wrap small amounts of Objective-C so that it can be used in C++. I have opted to keep Objective C out of the general C++ not just to properly isolate the complete madness of two different incompatible types exception handling but also to keep things simpler. It is generally natural to express these as C++ methods

QTKit and boost

I have started working on sound. Somewhere close to he start of the project I focused on getting the project to compile. The support for WAV files butted straight on to MFC so I compiled it out with a #define _NO_WAV. What it means is to work through this code I can remove the #define and work through the compile errors.

The support for WAV files in Cello is quite simple. WAV is converted to PCM using functions in MFC, and this PCM is then converted into the correct variant by some Cello code. So I need to be able to convert WAV (and other formats) to PCM. Trawling through Apple's sample code the way forward seems to be , as I suspected, to use QuickTime. I last used QuickTime, with any seriousness, about 13-14 years ago so it has changed a little.

The recent change is QTKit. QTKit is the Cocoa API for QuickTime. The new API introduces a new Cocoa object QTMovie that provides functionality like reading movies from a file. The first thing I need to do is mesh this into C++.

I could mix C++ and Objective-C without restriction but I would rather not. There are issues with two incompatible exception models - and I would like to keep the Cello engine as close as to the PC version as possible. Also I would like to write a version of the existing WAV handling class, which works, rather than to mesh in a new mishmash.

My approach, therefor, is to provide a C/C++ interface for the objects that I use, much like Apple have done with CoreFoundation. This is what I have been doing with some of the CoreFoundation objects that have richer Cocoa APIs than C APIs.

What I plan to do is to define a type for the Cocoa QTMovie as is done in CoreFoundation

typedef struct __QTMovie *QTMovieRef;

This means that I can use boost::intrusive_ptr to manage it

typedef boost::intrusive_ptr<__qtmovie> auto_QTMovie;
void intrusive_ptr_add_ref(QTMovieRef p);
void intrusive_ptr_release(QTMovieRef p);

And all I need to do is in the body of the implementation (the .mm file) to cast to a QTMovie

void intrusive_ptr_add_ref(QTMovieRef p)
{
 [(QTMovie *)p retain];
}

void intrusive_ptr_release(QTMovieRef p)
{
 [(QTMovie *)p release];
}

So an example function might look like this:

auto_QTMovie NewMovieFromUrl(CFURLRef url)
{
 NSError *theError = NULL;
 QTMovie *theMovie =
     [[QTMovie alloc] initWithURL:(NSURL *)url error:&theError];
 ThrowIfCFError((CFErrorRef)theError);
 return auto_QTMovie((QTMovieRef)theMovie, false);
}

Managing TextBreakLocatorRef using Boost

Since the birth of OSX new data types are based on Core Foundation. Core Foundation provides an owner-counted management of objects which allows for quite a natural, as well as simple, memory management. In C++ these are easily managed by boost intrusive_ptr (more details of which were posted here)

My initial take on TextBreakLocatorRef was that it predated Core Foundation and as I wanted to be able to manage these pointers using smart pointers I looked into how I might do this using boost and shared_ptr.

shared_ptr provides an owner counted reference to an object. The smart pointer is more complex than a simple pointer and manages an owner count. It is an very easy and reliable way to provide smart pointers to objects. The templated constructor of the shared_pointer permits the specification of a custom dispose (a custom deallocator). A reference to this deallocator is maintained by the smart pointer it's self.

So using shared_ptr you could manage TextBreakLocatorRef like this:
The only funny being that you would need to define opaquetextbreaklocatorref so that boost will consume it.

However investigating the issue further it seems that TextBreakLocator is managed by core foundation under 10.5. NSLog lists the locator as follows:
Consistency is always a good thing so I will use boost intrusive_ptr as I have done elsewere in Cello.

CoreText - Setting styles on empty AttributedStrings

[build 0058] Text in Cello appears to be completed - for the moment at least.

The last problem with text was something of a classic. Within Cello the text and style runs are stored as CFAttributedStrings. This works out really well as this is what CoreText consumes. The problem comes with empty strings. If you have an empty string then it is impossible to set any attributes on it. Also if you try and find the bounds of the string CoreText will give you an empty rectangle. This is not necessary what is wanted. If Cello is calculating the height of a text box then if text styling has been set then it needs to be able to set the styling of the text box and to use this to calculate the height of the box.

The solution to this problem is to introduce the idea of the NULL style. The NULL style is a style that is set and exists when there is a need to set the style at a point rather than over a selection of text. So, when there is no text, setting the style of a text flow will just set the NULL style. Then when text is actually added we grab the NULL style and style the newly added text with it. A similar concept can be used when you set the style at an insertion point.

As far as getting the height of text when thee is none - a solution that works is to give CoreText a single space character and to style it with the NULL style. Then the normal code that is used to calculate the box height will just work.

Core Text - CFStringGetParagraphBounds and Setting Paragraph Styles

I am working through the text classes writing unit tests. As Cello had no unit tests before I took on the OSX port I have been writing unit tests primarily for the delta, that is the changes I have made, rather than attempting to add unit tests for the all of the historic code. Sure I would love to but that is a project in it's self. I would hope that as I find bugs in the code (sure as are eggs are eggs there will be bugs) that I can add unit tests that demonstrate the bug and the fix after I have made it - but I will have to see.

I am using CFAttributedStringSetAttribute to add a paragraph style to the backing store. The only wrinkle here is that the style should be applied to the whole paragraph. Cello's pre Core Text code ensures that if you apply a paragraph style to a text range that the paragraph style will apply to complete paragraphs.

My hope was that CFAttributedStringSetAttribute with kCTParagraphStyleAttributeName would magically do this. It was a small hope and I quickly unvovered the folly of it when I wrote the unit test. Fortunately Core Foundation has a function CFStringGetParagraphBounds that will calculate the start and end points of a paragraph. The code falls out something like this.


CFRange selectedRange;
CFAttributedString attributedString;
CTParagraphStyleRef paragraphStyle;

// get the underlying string
CFStringRef str = CFAttributedStringGetString(attributedString);

CFIndex parBeginIndex, parEndIndex;

// get the get the bounds of the paragraph
CFStringGetParagraphBounds(str, selectedRange, &parBeginIndex, &parEndIndex, NULL);

// convert it to a CFRange
CFRange paragraphRange = CFRangeMake(parBeginIndex, parEndIndex - parBeginIndex);

// Set the attributes
CFAttributedStringSetAttribute(attributedString, paragraphRange, kCTParagraphStyleAttributeName, paragraphStyle);


CoreText and CTMutableParagraphStyle

I have implemented Cello's paragraph and character styles as thin wrappers for CTParagraphStyle and CFDictionary. This corresponds to the way that text styling is handled in core text.

The API for CTParagraphStyle is rather thin. There is, for example, no CTMutableParagraphStyle - which means that API supports the creation of a CTParagraphStyle in one hit - but you can not set properties one by one. Fortunately the CTParagraphStyle is toll-free bridged to NSParagraphStyle which has a far richer interface including the ability to make a mutable copy.

CoreText - the approach

I have been looking at the text in Cello in some detail. The approach that have decided to take is to change the fundamental text object so that it is a thin wrapper on a NS/CFAttributedString. Doing this gives me RTF import and Unicode support at a basic level. On top of this the CTFrame fits quite naturally into one of the Cello classes.

The fact that the PC version of cello is fundamentally based on RTF and that I can use core text as an RTF engine is a bit of a result. What it means is that I can deploy a unicode text editing engine (unicode character set, bidirectional text). This just has to be projected into the UI - but it a way basing things on core text and doing the UI in cocoa give or take the international input it is probably hard not to do.

The remaining piece in the multi-lingual text equation is vertical text - but this is not hard to do in core text - but it is not pressing.

Within Cello there are also classes that represent character and paragraph styles. I will try and preserve these but make them thin wrappers. The paragraph style would seem to map naturally onto the CTParagrahStyle, and the Character style as a NS/CFDictionary of values for the attributed string.

CoreText and RTF

Sample Unicode: 魚もたばません

Cello has text so it has support for line layout and line breaking. It has classes that represent Paragraphs, Styles, Style Runs and Layout blocks etc. Down at the metal the text is serialized (written and read) as RTF. Cello has its own RTF reader and writer. All this has to mesh with CoreText - that is where the glyph curves ultimately come from. Also in the back of my mind is the thought that I need to support Unicode, bidirectional line layout - and ultimately (I hope) vertical text.

I am still exploring the best course of action - but currently the solution that seems to be optimal is to use CoreText for everything.

Styled text is represented in Core Text as a NS/CFAttributedString. The NS/CFAttributedString is a Unicode string with runs of attributes. In the all roads lead to Rome approach of Cocoa and Core Foundation attributes are expressed as NS/CFDictionaries. So, for example, to set a font you would essentially create a dictionary with the kCTFontAttributeName attribute set to the CTFont that expressed the font that you wanted. To make the world a simpler place the API makes it possible to set individual attributes without having to create a dictionary first.

Cocoa also uses NS/CFAttributedString to represent styled text and has the capability to create an attributed string directly from an RTF source. So the following code fragment will read an RTF file from a URL.

CFAttributedStringRef ReadRTF(CFURLRef url)
{
NSData *data = [NSData dataWithContentsOfURL:(NSURL *)url];
NSAttributedString *result = [[NSAttributedString alloc] initWithRTF:data documentAttributes:NULL];
return (CFAttributedStringRef) result;
}

I modified Apple's CoreTextTest sample code to import RTF using the code sinippet above. The resulting CFAttributedString works fine in Core Text except that NSColor is not suported. The console fills with messages:

CGContextSetFillColorWithColor: invalid context
CGContextSetStrokeColorWithColor: invalid context

The reason behind this seems to be that cocoa attributed string uses NSColor, Core Text uses CGColor the two objects are not toll free bridged and are not the same. This does no look like a hard thing to fix up after reading, and before writing.

One of the advantages of using the Cocoa RTF import is that, unlike the Cello import, it supports unicode.

Here is a screenshot of this post saved as RTF being drawn by Core Text with a little unicode.

Reading Bitmap Objects I

Cello's file format is broken up into a number of distinct objects. The simplest the shape works and publishes correctly. All the object types are registered through a GUID and the read calls a factory to make objects that correspond to that GUID. This makes for an easy incremental development. All the registration *except* for the shape object is commented out. It is possible to uncomment them bit by bit and to work through them.

The plan is to work through the objects, register them, add the files that are required (working through compile and link errors) and then test.

Having registered the bitmap GUID and worked through the comparatively few errors and additional files required, and a crash in the Color Palette class bitmaps read and write. Publishing yielded a blank (missing) image. This is because the file format, under some circumstances" contains a reference to the image as a hard path. The fall-back behavior at publish time for a missing image is to just omit it.

Having a hard path to an image presents some problems for testing. the work around is to have a mechanism that will change the path in a file after the file has been been read. I did this by having a simple class that has a list of files with full paths and will match a file against the list based on the file name - so

/users/scratch/nonsense.jpg

will match against

C:\Documents and Settings\All Users\Sample Pictures\nonsense.jpg"

This kind of matching is easily done with Core Foundation's CFURL. CFURLCopyLastPathComponent will get the last path component of a path i.e. nonsense.jpg and URLs can be constructed form POSIX, HFS and Windows paths.

Having added the code - it is clear there are some problems with the publishing. See below:

DIB to CGImage - the washing up

I changed the plans and in the end kept the class that manage the DIBs almost unchanged. The DIB class, within Cello, represents a bitmap with a physical bit-image. Give or take the fact that it can handle native compressed bit-images it handles the concept that an image is a bunch of bits and possibly a color table. A CGImage is very similar except that you can't get to the bit-image. You can make a CGImage where you can get to the bits if wrap the the bit image in a CFDataProvider. So the natural thing seemed to keep the DIB class more or less as it was but to provide a way off creating one from a CGImage.

The key change that I have made to the DIB class is to change the way that pixels are stored; I have moved from using a raw pointer for the pixels to boost::shared_array. The boost::shared_array is an array with an owner count - and it means that it is possible to deal with joint ownership. The point being that data provider needs to have ownership of the data it is providing (the pixels) as does the DIB class.

The other part of the equation is the colour palette its self. A colour palette is a list of colours - little more. In Cello there is a class that represents this list of colours as a list of RGBQUAD from which it can create the HPALETTE etc required for MFC bitmaps. I have changed the colour palette class so that it is now a thin wrapper around a CGColorSpace. You can still access a representation of it as a list of RGBQUAD - this is required for other things in Cello such as the ditherer. Changing this code and writing the attendant unit test was quite straight forward. I use boost::intrusive_ptr to manage the CFColorSpaceRef, and boost::shared_array to manage representation (cache) of RGBQUAD. In retrospect I think it would have been better not to have made this change and instead to have just provided a way of converting into a CGColorSpace. It would have been a smaller change.

Implementing MFC CTime using CFDate

In the early stages of Cello I stubbed out the MFC CTime class with little more than a constructor in order to get it to compile. Working my way through the link errors this is something that I needed to flesh out.

My basic plan with implementing these MFC classes is to keep the API but to build them on top of Core Foundation objects. Core Foundation can quite ubiquitously be expressed as NS (Objective C) objects (they are toll-free bridged) which will pay dividends when I start working in cocoa for the UI. Also, and importantly, it is all done, it works and just requires hooking up.

I implemented the CTime class as a wrapper for a CFDateRef managed by a boost intrusive_pointer. Give or take a thin wipe of code and a modest set of Unit Tests that was it.

Implementing SetFileTime and GetFileTime under OSX

Working through the link errors makes for some catching up on some things that I sketched out and did not complete. I have been looking at the file handling class, file streams and writing unit tests to make sure that the they are working properly as I flesh them out.

I have backed Cello file handling class more or less straight onto STL file streams. In this class there are a couple of methods SetFileTime, and GetFileTime. In the MFC version these are more-or-less straight wraps of the API calls SetFileTime, and GetFileTime:

 BOOL WINAPI SetFileTime(
 __in      HANDLE hFile,
 __in_opt  const FILETIME *lpCreationTime,
 __in_opt  const FILETIME *lpLastAccessTime,
 __in_opt  const FILETIME *lpLastWriteTime
 );

Under OSX I have implemented them in terms of FSGetCatalogInfo and FSSetCatalogInfo. So, for example, the blood and guts of the SetFileTime method boils out like this:

 const char *path = (LPCSTR)m_dwsName;
 FSRef fsRef;
 Boolean isDirectory;
 FSCatalogInfo fsCatalogInfo;

 OSStatus err = FSPathMakeRef(reinterpret_cast(path), &fsRef, &isDirectory);
 ::ThrowIfOSErr(err);
 err = FSGetCatalogInfo(&fsRef, kFSCatInfoAllDates, &fsCatalogInfo, NULL, NULL, NULL);
 ::ThrowIfOSErr(err);

 fsCatalogInfo.createDate = *lpCreationTime;
 fsCatalogInfo.contentModDate = *lpLastWriteTime;
 fsCatalogInfo.accessDate = *lpLastWriteTime;

 err = FSSetCatalogInfo(&fsRef, kFSCatInfoAllDates, &fsCatalogInfo);
 ::ThrowIfOSErr(err); 

In my first cut I implemented FILETIME as a CFDate - however looking at FILETIME when writing the test cases I changed this to UTCDateTime which is actually practically the same.

The only wrinkle I have found is that the FSGetCatalogInfo ignore the fractional part in the UTCDateTime - which is something that tripped me in the Unit Tests

CString and Unit Tests

Right at the head of the link errors was the CString class and most of it's methods. Early on in the port I copied the class declaration from MFC, commented out almost all of the methods and then uncommented the ones that as I got compile errors. It is surprising just how few of the class methods are used. At the time I knew that I wanted to implement it as a CFString so I added constructors and accessors so that I could freely convert from CFString.

Implementing the CString methods was fairly straightforward - so much so that I paused before writing any UnitTests, not wanting to waste time.


I retire some hours later with renewed belief in writing unit tests. Many of the tests worked straight off the bat - but I found serious problems in my implementation of CString::Format that would have bitten me later and would have taken longer to fix at a later date. Also using the accesors to get the underlying CFString it has been possible to check that the owner counts of the CFString are being handled correctly - I am using boost::intrusive_pointer for the actual management - but the copy constructor and assignment operations should share the underlying pointers. Unit test identified some problems here.

There is an interesting post about unit testing by Will Shipley (Delicious Monster). As well as some good info about unit testing by Chris Hanson (where I found the link to the Will Shipely post).

Fonts and Font Scalars - from D-Type to CoreText

I have come to the part of Cello where I need to start dealing with text and fonts. Detailed information about glyphs, metrics, kerning pairs, glyph outlines etc. all need to be known in order to generate the final output file.

The Windows (MFC) version of Cello uses the D-Type font engine (scalar). Looking at D-Type it looks pretty good, and it is available for OSX. I do not, however, plan to use it - instead I will use CoreText. Core Text is Apple's new type technology. Released in Leopard (10.5) it replaces ATSUI which has entered the holding area that technologies enter before deprecation. I plan to use CoreText for a variety of reasons - the main one is that it integrates directly and naturally with CoreGraphics (Quartz) Apple's graphics technology. I plan to do all the drawing with CoreGraphics so using the complementary companion technology makes sense. I have experience of mashing and melding disperate graphics engines and font scalars and the like together and it is a whole lot easier to have a single imaging model.

The CoreText API looks clean and easy to understand. It is built on top of standard Core Foundation objects like CFDictionary as well as introducing it's own new types like CTFont. As with other Core Foundation technologies it toll-free-bridges onto equivalent cocoa NS types. This will make the inevitable protrusions into Cocoa for the UI straightforward to deal with.

Boost - Core Foundation

I have started plumbing in some of Core Foundation into Cello. Core Foundation objects are owner-counted which leaves you to take care of calling CFRetain and CFRelease at appropriate moments. It is very straight forward except that adding all of the CFRetain and CFRelease clutters your code and (more importantly) can be tricky to get right - it is really easy to forget a release, especially if you consider exceptions. .

Things are eased a little if you have classes that wrap some of the Core Foundation objects - but the thing that really makes the difference is smart pointers.

In a previous life I used Apple Class Suites (ACS) which was a part of MacApp. ACS wrapped most of the underlying OS APIs in good C++ and introduced me to using smart pointers that would automatically call CFRetain and CFRelease. So if you wanted a pointer to a CFString that would automatically call CFRelease when it went out of scope there was an AutoCFString_AC smart pointer.

There is excellent support for smart pointers in boost. Declaring and implementing a smart pointer using intrusive_ptr to do the same as MacApp's AutoCFString_AC takes just three lines of code in boost:

 typedef boost::intrusive_ptr auto_CFString;
inline void intrusive_ptr_add_ref(CFStringRef p)
   { ::CFRetain(p); }
inline void intrusive_ptr_release(CFStringRef p)
   { ::CFRelease(p); }

For those curious in the MacApp annex of history, MacApp though 'dodo dead' has played an important part in the development of many application arameworks, there is an insightful article in wikipedia here .

Preferences and files that will never be ported

While working through the first set of Cello utility files I came across a couple of files - the first was DWNamedTreeStorage an XML reader and writer class. I fixed the compile errors (small string-type stuff) and moved on. It occurred that I could implement whatever there is in the file with some of the Core Foundation which has good support for reading and writing XML. However that is deviating from the game-plan. The game-plan is to get a subset (just reading files and generating the output file) as quickly as possible. Everything in Cello running on Windows works and it works well - there are many other wheels looking for reinvention.

Following this was a file called DWNamedTreeStorageRegistry. There was a fair amount in this file that would not compile and much of it was very OS specific - stepping back and looking where it was used - I found that it is used to save and load preferences from the registry. I have moved this file off the hit-list and will implement it's functionality after using CFPreferences when I come to the client code that uses it. It may well be that DWNamedTreeStorage is never used.

To keep a record of files that will never be ported - but whose functionality will have to be otherwise implemented I have added a text file called 'BadList' to each library.

CVS and Cutting the first Strings

I have created a new CVS repository and checked in the original sources. CVS is not the best control control system but it is not bad and does the job. I had at the outset thought about getting the project professionally hosted - there is some good Trac SubVersion hosting out there - but for the moment I will save the money.

I have started working through some of the utility classes and getting them to build. The code is well organized, structured and generally well commented. Coming from a long legacy project which in reality is a bit out of control working with code like this is a real breath of fresh air. I was fortunate enough to have spent a day looking at the code before buying into it, during that initial review I could see that it was well written and of a high quality. The most daunting aspect of it all is the shear quantity - there are hundreds and hundreds of files.

My first step is just to get things to compile. There are a few fun and games with templates as gcc is quite strict. There are (obviously) a lot of types defined in MFC - things like BYTE DWORD etc. I am making some stub headers with the minimum of declarations in them.

I have taken the MFC CString class and turned it into a wrapper for a CFStringRef - the code has it's own string class and it occurs to me that down the road I might consolidate them. The windows version of Cello is not Unicode which is something that down the line I will want to change. Making CString a wrapper for CFStringRef is a start - and those parts of the interface to CString that are char * I am scoping in a #ifdefs so that down the line I can flip the define, have my build break and work through the the compiler errors. The other thing is that the CFString is when I come to the UI a NSString - ready for cocoa.

I have after a good days work got 8 files compiling - the first took half a day but it gets quicker!