The end of sound

[build 0114] I have reached the end of sound - so the files with sound (WAV and MP3) generated on the PC open and publish correctly.

The next step is the user interface. However before that I will go back over the new code I have added and do a little cleaning and re-factoring. I am not planning anything big - just making sure that things like names are consistent, that the code is commented and documented. This sort of thing, though it takes time, pays off in the long run.

Sound - nearing the end

I think I am nearing the end of sound. I have created a new class that is heavy based on Apple's SCAudioCompress sample code and the existing MFC WAV to PCM converter class. So what I have got is more-or-less a drop in replacement for the old class.

The approach I took was to take Apple's SCAudioCompress sample code and transformed it bit-by-bit, into the form I wanted, taking fairly small steps. I took the code that I had crudely hacked into place and transformed it into the Class I wanted and properly integrated it into Cello. While transforming I continually tested the code making sure that it worked after the transformations - I did this by creating a number test files with sounds. I used some of the sounds at Wave File Sample page as samples.

I am not quite there one or two transformations left, some testing and documentation - but my guess is that it should, I hope, fall out in a couple of evenings.

Sound - Debugging

The biggest drawback for me when working with sound is that I don't know that much what I am doing. What I mean is that I have spent many years, for example, working with graphics, transformations etc. and I know the language, terminology and have encountered many problems. The upshot of this is that when working with graphics I have a context to put things into and looking through the code it "makes sense", as do the problems. With sound everything is new. In this regard OSX is basically as unfamiliar as MFC.

My one good stoke of luck is, as ever, that the code is beautfully written and well commented. From the code I know that WAV files are converted to PCM. These files are then consumed by Cello's own ADPCM converter and appear in the final output file.

Having looked through Apple's sample there is a sample SCAudioCompress command line tool that converts sounds. It also has an optional UI that means it is posible to play around with settings. Hacking this in to a class that was based on the original WAV source was not hard. My approach was to just get it in to see if I was on the right track. Once I had something working there would be an opportunity to refine.

My first test file was a short movie with a phrase from Peter Griffin (from "Family Guy").

Adding the code was not too bad. The existing tool wrote a file and loading a buffer wit the contents of a file is quite straight forward. The results were disappointing - the first sound that Cello published was short and unintelligible. Not understanding much about sound I felt a certain amount of despair - where to start?

To debug it I compared the two builds in the debugger. Running Visual Studio under VMware means that I can look at the two builds side-by-side. I also compared the two log files that are created during publishing. My discovery was that the OSX build outputted roughly a 10th of the sound than the PC build. The bug was that in calculating the duration of sound I was out by a factor of 8 - the difference between 8 bits and 1 byte. After fixing this the sound appeared. The unintelligibility of the first part of the sound was due to he strange form of Peter Griffin's voice. If I had chosen a voice like Brian - my life would have been less fraught.

QTKit and boost

I have started working on sound. Somewhere close to he start of the project I focused on getting the project to compile. The support for WAV files butted straight on to MFC so I compiled it out with a #define _NO_WAV. What it means is to work through this code I can remove the #define and work through the compile errors.

The support for WAV files in Cello is quite simple. WAV is converted to PCM using functions in MFC, and this PCM is then converted into the correct variant by some Cello code. So I need to be able to convert WAV (and other formats) to PCM. Trawling through Apple's sample code the way forward seems to be , as I suspected, to use QuickTime. I last used QuickTime, with any seriousness, about 13-14 years ago so it has changed a little.

The recent change is QTKit. QTKit is the Cocoa API for QuickTime. The new API introduces a new Cocoa object QTMovie that provides functionality like reading movies from a file. The first thing I need to do is mesh this into C++.

I could mix C++ and Objective-C without restriction but I would rather not. There are issues with two incompatible exception models - and I would like to keep the Cello engine as close as to the PC version as possible. Also I would like to write a version of the existing WAV handling class, which works, rather than to mesh in a new mishmash.

My approach, therefor, is to provide a C/C++ interface for the objects that I use, much like Apple have done with CoreFoundation. This is what I have been doing with some of the CoreFoundation objects that have richer Cocoa APIs than C APIs.

What I plan to do is to define a type for the Cocoa QTMovie as is done in CoreFoundation

typedef struct __QTMovie *QTMovieRef;

This means that I can use boost::intrusive_ptr to manage it

typedef boost::intrusive_ptr<__qtmovie> auto_QTMovie;
void intrusive_ptr_add_ref(QTMovieRef p);
void intrusive_ptr_release(QTMovieRef p);

And all I need to do is in the body of the implementation (the .mm file) to cast to a QTMovie

void intrusive_ptr_add_ref(QTMovieRef p)
{
 [(QTMovie *)p retain];
}

void intrusive_ptr_release(QTMovieRef p)
{
 [(QTMovie *)p release];
}

So an example function might look like this:

auto_QTMovie NewMovieFromUrl(CFURLRef url)
{
 NSError *theError = NULL;
 QTMovie *theMovie =
     [[QTMovie alloc] initWithURL:(NSURL *)url error:&theError];
 ThrowIfCFError((CFErrorRef)theError);
 return auto_QTMovie((QTMovieRef)theMovie, false);
}

Canned Effects - complete

[build 0096] I have completed working through all of the "Canned Effects", including the text effects, and they now work under OSX. Once I had completed the various problems with the handling of text it all fell out reasonably quickly. Getting the effects to work was little more than a handle turning exercise - get the files to compile, generate test files and test.

The next step is to support files with sound. My guess is that I will need to use QuickTime - this is more of a hunch than the result of serious analysis.

Understanding Fixes - Debugging

As I work trough the text effects I make test files on the PC version so I can test on the OSX version. The test files are reasonably straight forward - I test the range of settings, use different styling and a mixture of paragraph and character styling. In this I came across a problem. The issue was a word based effect with a change of style within the word resulted in some characters being badly positioned. The screen shot below is an example is the output from the simplest case I could create that demonstrated the problem - the "r" is kicked down below the base-line.
This bug had me floored - it was just very hard to work out why it was not working. Running the PC and OSX versions back to back can be very helpful but the in the case of text and fonts there are a lot of numbers many of which are different on the two platforms. CoreText , for example, gives different ascent and descents for fonts. Much of the math in Cello is integer and CoreText is fully floating. As an example of some or the sorts of differences you can get [even] Arial is different on the PC than then Mac. As an example below is a comparison between the same glyph in Arial on the two different platforms.

Getting nowhere for a while I wrote a list of all the possibilities that might bet the cause problem and worked through them. This got me as far as the point where the offset for positioning a glyph within an effect block was calculated. I discovered that if I abandoned the offset calculation, for the glyph that was failing, and set the offset to 0 (so it was the same offset the other glyph) everything worked!

OK- so this was a natural time to give up the chase and move on - but it really niggled me. I had no idea why forcing the offset to zero got things to work. I could kind of imagine that the offset might be zero in the PC version, or that CoreText was giving me values that were different enough for somehow this to be right or something. Checking on the PC version there was an offset that was different from mine but definitely not zero and not that far away.

Moving on and saying "well sure it works now" was tempting. I have worked for many years with a really excellent programmer who is careful, methodical, and very good. When things happen and start mysteriously to work wants to know why. He also, incidentally, writes some of the best code I have seen. I have learnt a lot from him - I carried on the chase. This morning before work I found the problem - misunderstanding some of the code I was not aggregating separate styles within a common word into the same effect block. This was an important bug to find and my setting the offset magically to zero would have been a bodge that would just have masked the problem.

Text Effects - Debugging rounding errors

The first of the text effects is an effect where glyphs are swapped our randomly. Having fixed the more obvious (large scale) problem in the positioning (as described here) there was an outstanding more subtle bug in the positioning of the effect glyphs.

One of my test files is based on some effect text that is precisely positioned on top of some normal text. When the effect ends the text come to rest so it exactly covers the normal text. In the screen shot below you can see that there is a vertical and horizontal offse that looks a bit like a drop-shadow.These offset problems were, as I suspected, due to rounding errors. The PC version of Cello deals exclusively with integer values when positioning text. CoreText, on the other hand, deals with floating point values. The output format deals with integer TWIPS. The issue with the effects is that Cello carries about bound rectangles for the effects (in this case glyph-by-glyph) that are stored as integers before being multiplied by 20 to become TWIPS.

Once I uncovered the problem (it took a while to work out where it was going wrong) I replaced the CRect (a MFC integer rectangle) with a floating point equivalent. Ultimately I need to go through all of the text calculations and make sure that floating point is used - but this is something for my to-do list rather than for now. At the moment I want things to work - but also the minimum of disruption in getting the effects to build and work.

Text Effects - Debugging bounds problems

I have been looking at a strange problem where glyphs, when revealed by an animation, fail to redraw properly. What happens is when a part of an animation moves over a character shape the shape that is revealed sometimes fails to redraw. You can see this in the snapshot below - most obviously there is a bit missing from the A.


When I first saw the problem I presumed that it was down to a bug in the player and dismissed it as "just something that happened" however as time has gone on I have noticed other situations where the same problem. Going back to the PC version of Cello I can see that the problem does not exist - so obviously it is a problem that I have introduced.

I must admit to being daunted when I first looked at the problem - I had little idea why the problem would happen, but like most bugs in the end it was not as bad as I thought. The key things are to take the time, think it through, and get a plan that will at the very least narrow the problem down.

The example animations I was working with were, in terms of the output, quite complex. The examples had a fair handful of glyph shapes. The first thing I did was to create a sample that was as simple as possible. I found I could get the problem to occur with just two characters that were on two different lines. I could also further simplify the example by using an effect on simple shape that was on top of the characters. If I made the characters the same this was a further simplification. These simplifications were important - the type of the shape moving seemed to me unimportant - the problem seemed to be in the glyphs that were being revealed.

The next part of the debugging was to compare the output against that of the PC version. The Engineer who originally wrote Cello did a really good job. Amongst the things he did was to add in debugging code that would parse the resulting file and output it as text. What this means is that I can run the PC version of Cello, then the OSX version and diff the two files. I have done this in the past when working on glyphs and text and it is invaluable.

Looking at the diff of the file I found a few things. The PC version of Cello had an extra redundant curve segment - effectively a moveto(0,0). I changed the code so it was added. The OSX version output an empty glyph (for the so I changed the code to stop this from happening). Working through the diff the next thing was the defineTextTag - the bounds were different - and looked wrong. And this turned out to be wrong. My basic error was in constructing a CGRect I hat the origin of the Rect at the top rather than the bottom. The player will happily draw outside the bounds - but it respects the bounds when calculating a redraw area. Seems reasonable enough to me.

Text Effects - Debugging

Having implemented the Accessor class getting the base class and the first text effect to compile was reasonably easy. The first text effect is a glyph substitution effect where glyphs in text are replaced with random glyphs in a simple animation.

After fixing a couple of crashes the output appeared to more or less work. To test the effect more thoroughly I overlaid the text with the effect applied (a single word) on top of the same text without the effect — this allows me to check things like positioning and character spacing. Doing this and publishing revealed that the text was positioned wrongly and had a vertical offset — so there is a bug.

After quite a few years of chasing bugs a method that often really helps me is to stop and think about it and see if you can figure out how it is misbehaving. Are there different — maybe better test cases that will tell you more about the problem.

So on to this offset — is it a one off offset (say the position of the text box) that is causing the problem or is it something that is incorrect with the calculation of the baseline. Knowing this will allow me to more tightly narrow my focus. There are a lot of numbers flying about in Cello and being able to look in the right area is going to be much less painful.

To test this I made two lines of test (two paragraphs with one word) in the PC version of Cello and published. If you look at the results below:



So from this it looks like the offset increases with successive lines. My guess is that the problem is somewhere in the calculation of the baseline or bounding rectangle of the text — both of which have logic to flip from CG style coordinates to QD style coordinates.

Less is More

The observation Less is More is an observation of Cello project that comes home time and time again. A considered path of least resistance seems generally to be the best course.

For the past week or so I have been working on the text effects and I had sort of stalled. Cello is an after hours project so it is hard to get a stretch of more than a few hours together. The text effects stalled because of the complexity of reworking the effects base class so that it would deal withe a CoreText view or the world (CTLine, CTRun, CFGlyph, CFString) rather than the inbuilt text. Ultimately if it takes a cold towel on your head to figure out what to do something somewhere is probably wrong.

Access to the old pre-CoreText text was made through an Accessor class that iterated through lines and style runs. My mistake was to not use or implement it. I went straight to CoreText for the text output - it seemed right I needed to add the distinction between glyphs and characters. The crunch was that the Accessor class is a component that is passed about for the text effects.

The better solution is instead to use and re-implement Accessor class so it sits on top of CoreText and change it so that it can handle glyphs. This is not hard to do and essentially preserves the way that text effects are written.

My approach here is to write the new Accessor class and go back to my text export (which works) and deploy it. I have good test files and an understanding of how it should work. I can also shape the Accessor class in place and round it in place if necessary. Then I will go back to the text effects base-class and plumb in something that may have a different thread but is of the same bore.

Managing TextBreakLocatorRef using Boost

Since the birth of OSX new data types are based on Core Foundation. Core Foundation provides an owner-counted management of objects which allows for quite a natural, as well as simple, memory management. In C++ these are easily managed by boost intrusive_ptr (more details of which were posted here)

My initial take on TextBreakLocatorRef was that it predated Core Foundation and as I wanted to be able to manage these pointers using smart pointers I looked into how I might do this using boost and shared_ptr.

shared_ptr provides an owner counted reference to an object. The smart pointer is more complex than a simple pointer and manages an owner count. It is an very easy and reliable way to provide smart pointers to objects. The templated constructor of the shared_pointer permits the specification of a custom dispose (a custom deallocator). A reference to this deallocator is maintained by the smart pointer it's self.

So using shared_ptr you could manage TextBreakLocatorRef like this:
The only funny being that you would need to define opaquetextbreaklocatorref so that boost will consume it.

However investigating the issue further it seems that TextBreakLocator is managed by core foundation under 10.5. NSLog lists the locator as follows:
Consistency is always a good thing so I will use boost intrusive_ptr as I have done elsewere in Cello.

Unicode Word Breaks - UCFindTextBreak

Working through the various effects I have come quite quickly to text - so the holiday (the appearance of rapid progress) is over and it is back to text. I like text so all is right with the world.

The text effects work on either lines of text, words within lines or letters. In the PC version of Cello there is no distinction between a letter and a glyph - however gunning everything through CoreText things are a bit different and there is this distinction to deal with.

The issue of word breaks is also more complex if you are dealing with Unicode. The PC version of cello deals only with Roman text and the word breaks are based on white-space. This does not hold for all languages and character sets - for example Japanese words are not bounded by white-space so this method of detecting word breaks (obviously) won't work in the general case.

A solution to the detection of word breaks is to use Apple's Unicode Utilities which has a Text Break Locator that discover text breaks. The format is reasonably simple UCCreateTextBreakLocator creates a text break locator, UCFindTextBreak will then locate a text break in a run or UTF16 characters.

The amount of wriggeling to get UCFindTextBreak to work with a CFString is very modest.

Core Text - CTFontGetUnderlineThickness

[build 0066] I have got the first five effects to work - that is I can open files that have the effect and publish them. My thought of a chance to make some rapid progress by avoiding text was short-lived. Having added text to the test files there were problems to work through.

The effects deal with text as path shapes. That is internally they make a single shape out of the glyphs and then do weird and wonderful things to them. This exercises a section of the text code that I had identified as not being tested and was open on my check-list (I could not generate file that was able to test it). The primary problem was due to incorrectly scaling the glyph shapes as they were generated, but there were a few others.

Working through the bugs the last one was the position of the underline. Below is an effect that transitions a shape my curling it upwards. In the test file je orange text starts exactly over the bluish text and curls off. It is caught part way through transition.

Interestingly a similar problem affected the PC version of Cello - the underline shifted once an effect was applied. The two underlines are calculated in two different place and calculated differently.

The solution was quite nice. CoreText has two functions CTFontGetUnderlineThickness and CTFontGetUnderlinePosition - just call these functions in the two places and the results are identical.

Canned Effects

[build 0062] Having got to the end of Text I found that Sprites, Gradients, Buttons and TimeLines fell out in a few hours. In some ways this is strange because progress has been pretty hard fought for with text. In getting the text to work I had to replace a font scalar, plumb in CoreText - none of this is small beer. The Sprites, Gradients etc are little more than getting the files to compile - registering them with object factory and testing, Later, when it comes to writing UI the shoe will be on the other foot.

The next step for Cello is getting the various canned effects to work. The canned effects are animation effects that are applied to an object in a single hit and just "work". The effects come in different classes. Logically I should probably have attacked the text effects first as they would mesh into the text and probably would lift out any remaining bugs in the area. However having done two months of text I have decided to start with some of the other effects first. My thought was it would be well, just nice, to appear to make a little progress. So I have started the effects in alphabetical order.

The first effect is an effect that takes a shape and maps it onto all four sides of a spinning cube. Getting the files to compile (there were nine new files) took two or so hours - and afterwards it well almost worked - just objects are wrongly positioned and spin, well, strangely.

The reason made me smile. I had to add an implementation of CenterPoint to my MFC CRect class. The function was so trivial that I did not write a unit test. I was lucky to spot the error when I did a brief code review before checking my changes into CVS.

CoreText - Setting styles on empty AttributedStrings

[build 0058] Text in Cello appears to be completed - for the moment at least.

The last problem with text was something of a classic. Within Cello the text and style runs are stored as CFAttributedStrings. This works out really well as this is what CoreText consumes. The problem comes with empty strings. If you have an empty string then it is impossible to set any attributes on it. Also if you try and find the bounds of the string CoreText will give you an empty rectangle. This is not necessary what is wanted. If Cello is calculating the height of a text box then if text styling has been set then it needs to be able to set the styling of the text box and to use this to calculate the height of the box.

The solution to this problem is to introduce the idea of the NULL style. The NULL style is a style that is set and exists when there is a need to set the style at a point rather than over a selection of text. So, when there is no text, setting the style of a text flow will just set the NULL style. Then when text is actually added we grab the NULL style and style the newly added text with it. A similar concept can be used when you set the style at an insertion point.

As far as getting the height of text when thee is none - a solution that works is to give CoreText a single space character and to style it with the NULL style. Then the normal code that is used to calculate the box height will just work.

CoreText Typography - Ligatures

[build 0051] The great fight with text is slowly coming to a close. I have a few outstanding bugs before I move on to editable text but I can see the end.

Ligatures appear correctly in the output.
Having spent some time looking into glyphs and glyph codes ligatures are the easiest test case for non-standard glyphs because it is possible to type in normal letters like "f" and "i" in the PC version of Cello, save the file, open them under OSX and then to let the CoreText layout transform the pair into a ligature during normal line layout. My conjecture is that the treatment of ligatures will prove a reasonable test case for non-standard glyphs (e.g. various alternate character forms) in general. As Cello shapes up more it will be possible to test this further.

If you are interested in typography here is a link to SIL International has some good information here.

CoreText - Accesing the CMAP CTFontGetGlyphsForCharacters

CoreText provides the function CTFontGetGlyphsForCharacters that will give you a conversion from a letter to a glyph through the unicode CMAP.

Glyphs to Character Codes - revisited

My original plan to map glyphs to character codes appears doomed. There main flaw in this plan is that it looks like the glyph names are not reliable - what I mean by this is that some glyphs do not have names that can be mapped to a character code. For example the font "Li Song Pro" has glyph names names that apper to be based on glyph codes rather than character codes.

At this point I took stock what I was trying to achieve and why the character codes are necessary. Character codes are required because there are occasions in the output where a font is output that is "complete" with a correct character mapping is required. To satisfy this all the characters that are part of the "normal" mapping must be present - as well as any additional character that may be used by specific text in the output (fonts are shared across objects). Other parts of the output requires partial fonts - that is just the glyphs in a font that are actually used. In this case the character codes don't really matter. As there can be overlap when glyphs are output if they are a part of the "Standard character set" then they should be identified in order that they can have the correct character codes attributed to them.

Looking at things in this way it simplifies things. We need to identify if glyphs are a part of the standard character set. The solution that seems to present it self is this:

1. Locate the character(s) resonsible for a paticular glyph
2. Check if that character is a part of the stabdard character set - this can be done by mapping the character to a glyph code using the font's CMAP and checking if we get the same glyph code.
   

CoreText - Getting Glyph Names

Looking at the published file and its log I can see that there is a problem with the character codes that Cello is publishing. The published file format expects character codes that are then mapped to glyph indexes. Cello is feeding the code glyph indexes, it works but it is not right and will cause problems in other places where text is used. If you are not too familiar with the difference - Wikipedia has an excellent introduction here.

Getting glyph codes as a result of line layout is a natural consequence of using Core Text. Core Text takes styled text and turns it into glyph runs (lists of glyph codes) - this is exactly what I want. Core Text does all the clever bits of positioning, ligatures, alternate character sets etc. So the problem that I face is the need to convert glyph codes back to character codes.

At this point it is worth mentioning that it is not always possible to convert glyph codes back to character codes. Line layout will do things like convert some some multiple character forms to ligatures - see. Also some fonts can have different forms - for example "end swashes" where a character at the end of a line can be represented with different glyphs, and it goes on. The best you can hope for is to be able to convert most glyph codes to character codes.

I have been thinking about how to approach the problem. The best solution that I can see at the moment is to

1. Pull the name of the each particular glyph - Core Text does not support this directly but it is possible to convert a CTFontRef to a CGFontRef using CTFontCopyGraphicsFont, and then getting the name using CGFontCopyGlyphNameForGlyph.
   
2. Using Adobe's magic technique map each glyph onto a character code as described here.
   

CoreText - Bold, Italic and CTFontCreateCopyWithSymbolicTraits

[build 0047] The text publishing is slowly falling into place. I have managed to get to the bottom of most of the positioning problems. The side-by-side comparison shows publishing under OSX text is quite similar. The vertical positioning is a little different. Also under OSX the text is kerned - something that fell out from using Core Text.
More complex example (an unpleasent combination of styling) is also close.

There seems to be a bug that means that a small section of underline is missing - but more obviously some characters are not italic.

The lack of itallics is due to the way that Core Text handles fonts within a font family. A true italic font is actually a seperate font that has been designed and created specificaly by the font foudry, as is a bold version of a font. If you ask for the bold or italic form of a font (CTFontCreateCopyWithSymbolicTraits and friends) Core Text will look for an approriate font in the family - if there is not one it will fail. This is typographically the correct thing to do.

Many programs will synthesize an italic font by slanting the regular font - this works but looks a little ropey. This is what the PC version does. It is also posible to synthesize bold forms of a font. This is done by out setting the contours of each glyph. If a synthesized italic font is a bit ropey a synthesized bold tends to be a bit ugly. It is quite possible to do both in Core Text but I currenty do not plan to do it.

CoreText - flipped coordinates and glyph curves.

Text output is requires partial fonts to be included in the file format. This is done by having the glyph outlines embedded in the file. As described earlier I am using Core Text to get the glyph outlines (see). Unfortunately (looking at the output) the outlines are flipped.

This particular problem is caused by CoreGraphics (Quartz) having a flipped y-axis, so zero is at the bottom rather that the top. This can be corrected by passing an appropriate transform (flipping allong the baseline) through to CTFontCreatePathForGlyph. This snippet calculates an appropriate flipped transform.

sFlipTransform = CGAffineTransformIdentity;
sFlipTransform = CGAffineTransformScale
(sFlipTransform, 1, -1);

Which results in the glyphs being correctly orientated.

CoreText and Typographic Bounds

[build 44] Having got the simple file with text to open the next stage was to get the brute to publish. The first attempt at publishing yielded nothing - white space. Looking at the debug log that dumps the published movie it was because the text object that was published was empty i.e. had no text.

The primary cause of this was due the conversion of floating point to integer values. Within Cello all object coordinates and dimensions are expressed as integers - Core Text, like CoreGraphics, is floating point. The issue here is that Cello has auto hight boxes whose height is based on the typographic bounds of the text. So a height of 31.2 was being truncated to 31. Giving a CTFramesetter anything less than the typographic height of the first line when it creates the CTFrame results in a frame with no lines of text. So no text. I found a report of a similar problem here.

Following a fix to a crash in some very early asci->uft8 code I wrote before I started writing unit tests Cello publishes it's first text.
The glyphs are not in the right positions and everything is inverted (coordinate space in Quartz is flipped so this kind of problem is not uncommon or unexpected) but it is not that far away.

CoreText & Cocoa Text - what is RTF?

[build 42] Cello will open some simple files with text, The question "what is RTF" seems suddenly quite poignant. The reading of a file with text fails because the reading of the RTF (within the file) fails. Cocoa believes the RTF to be bad and throws up it's hands in horror. Cross checking my results with TextEdit (I added some code to squirt out the RTF on its own in a file) yields the same result.


My assumption was that Cello's RTF could be read by the Cocoa RTF reader. Lawrence Harris (one of the most notable and helpful contributors to Apple's Carbon Dev mailing list) frequently points out the fallacy of relying on assumptions. The relevant RTF snippet is happily gobbled up by MSWord so it can't be all that bad, but my assumption was.

I spent some time thinking of the best way to deal with this. My feeling is that I need to be able to read files from the PC version to be able to do any kind of testing, my approach relies on working through files created on the PC version. My solution is to take Cello's RTF reader out of the "bad-list" and to use it as a fall-back for reading RTF.

Getting the RTF import to work was not a huge amount of work- probably the biggest surprise was the discovery that all text sizes in cello are represented in 10ths of a pt. Other than this I compiled out the inbuilt RTF export - this is currently not required and getting it to compile will just take time.

Text - the first test and the first crash

[build 41] I have got as far as linking in the text related files for reading, writing and publishing. This means that all the code that is required to read a text object, write it, and publish it has been added to Cello. As my first test I have created a text file in the PC version of Cello with the single word "Text".


Attempting to open this crashes - this, of course, is progress. I have isolated the crash and added it to the unit test which also crashes. This means that when I have the fix I can cast the fix in 'stone'.

Core Text - CFStringGetParagraphBounds and Setting Paragraph Styles

I am working through the text classes writing unit tests. As Cello had no unit tests before I took on the OSX port I have been writing unit tests primarily for the delta, that is the changes I have made, rather than attempting to add unit tests for the all of the historic code. Sure I would love to but that is a project in it's self. I would hope that as I find bugs in the code (sure as are eggs are eggs there will be bugs) that I can add unit tests that demonstrate the bug and the fix after I have made it - but I will have to see.

I am using CFAttributedStringSetAttribute to add a paragraph style to the backing store. The only wrinkle here is that the style should be applied to the whole paragraph. Cello's pre Core Text code ensures that if you apply a paragraph style to a text range that the paragraph style will apply to complete paragraphs.

My hope was that CFAttributedStringSetAttribute with kCTParagraphStyleAttributeName would magically do this. It was a small hope and I quickly unvovered the folly of it when I wrote the unit test. Fortunately Core Foundation has a function CFStringGetParagraphBounds that will calculate the start and end points of a paragraph. The code falls out something like this.


CFRange selectedRange;
CFAttributedString attributedString;
CTParagraphStyleRef paragraphStyle;

// get the underlying string
CFStringRef str = CFAttributedStringGetString(attributedString);

CFIndex parBeginIndex, parEndIndex;

// get the get the bounds of the paragraph
CFStringGetParagraphBounds(str, selectedRange, &parBeginIndex, &parEndIndex, NULL);

// convert it to a CFRange
CFRange paragraphRange = CFRangeMake(parBeginIndex, parEndIndex - parBeginIndex);

// Set the attributes
CFAttributedStringSetAttribute(attributedString, paragraphRange, kCTParagraphStyleAttributeName, paragraphStyle);


CoreText, RTF, Unit Testing

I have added the reading and writing of RTF, using the initWithRTF and RTFFromRange cocoa methods. These deal with NS/CFAttriibutedStrings, and back onto NS/CFData. The wrinkle is that NSColor has to be converted to CFColor as I covered before here.

The accompanying Unit Tests were quite simple, Create a CFAttributedstring, add attributes, write as a CFData, read back and compare. They should be equal.

The great thing about unit testing is that it throws up problems - though sometimes I prefer the older ostrich ways. I got the full twelve rounds on his one even the crash just after the test harness says everything is OK and tries to shut shop for the night. The crash turned out to to be an owner-count problem, once the owners drop below zero who owns what, and what is what, is a little ambiguous. I remember in my student days having this kind of ownership ambiguity with the bank - exactly whose money was I spending, and why did they get so upset.

Probably the most unexpected problem was in hindsight the most obvious. If you specify a color as say rgb(0.1, 0.2, 0.3) it comes back from RTF different. there are two parts to the problem. The first is color space (or profile). Colors went in as device color and came back as a NSCalibratedRGBColorSpace. Converting between color spaces (or profiles) changes color values. The other is that RTF stores color values as 0..255 - and of course naturally you loose precision.

CoreText and CTMutableParagraphStyle

I have implemented Cello's paragraph and character styles as thin wrappers for CTParagraphStyle and CFDictionary. This corresponds to the way that text styling is handled in core text.

The API for CTParagraphStyle is rather thin. There is, for example, no CTMutableParagraphStyle - which means that API supports the creation of a CTParagraphStyle in one hit - but you can not set properties one by one. Fortunately the CTParagraphStyle is toll-free bridged to NSParagraphStyle which has a far richer interface including the ability to make a mutable copy.

CoreText - the approach

I have been looking at the text in Cello in some detail. The approach that have decided to take is to change the fundamental text object so that it is a thin wrapper on a NS/CFAttributedString. Doing this gives me RTF import and Unicode support at a basic level. On top of this the CTFrame fits quite naturally into one of the Cello classes.

The fact that the PC version of cello is fundamentally based on RTF and that I can use core text as an RTF engine is a bit of a result. What it means is that I can deploy a unicode text editing engine (unicode character set, bidirectional text). This just has to be projected into the UI - but it a way basing things on core text and doing the UI in cocoa give or take the international input it is probably hard not to do.

The remaining piece in the multi-lingual text equation is vertical text - but this is not hard to do in core text - but it is not pressing.

Within Cello there are also classes that represent character and paragraph styles. I will try and preserve these but make them thin wrappers. The paragraph style would seem to map naturally onto the CTParagrahStyle, and the Character style as a NS/CFDictionary of values for the attributed string.

CoreText and RTF

Sample Unicode: 魚もたばません

Cello has text so it has support for line layout and line breaking. It has classes that represent Paragraphs, Styles, Style Runs and Layout blocks etc. Down at the metal the text is serialized (written and read) as RTF. Cello has its own RTF reader and writer. All this has to mesh with CoreText - that is where the glyph curves ultimately come from. Also in the back of my mind is the thought that I need to support Unicode, bidirectional line layout - and ultimately (I hope) vertical text.

I am still exploring the best course of action - but currently the solution that seems to be optimal is to use CoreText for everything.

Styled text is represented in Core Text as a NS/CFAttributedString. The NS/CFAttributedString is a Unicode string with runs of attributes. In the all roads lead to Rome approach of Cocoa and Core Foundation attributes are expressed as NS/CFDictionaries. So, for example, to set a font you would essentially create a dictionary with the kCTFontAttributeName attribute set to the CTFont that expressed the font that you wanted. To make the world a simpler place the API makes it possible to set individual attributes without having to create a dictionary first.

Cocoa also uses NS/CFAttributedString to represent styled text and has the capability to create an attributed string directly from an RTF source. So the following code fragment will read an RTF file from a URL.

CFAttributedStringRef ReadRTF(CFURLRef url)
{
NSData *data = [NSData dataWithContentsOfURL:(NSURL *)url];
NSAttributedString *result = [[NSAttributedString alloc] initWithRTF:data documentAttributes:NULL];
return (CFAttributedStringRef) result;
}

I modified Apple's CoreTextTest sample code to import RTF using the code sinippet above. The resulting CFAttributedString works fine in Core Text except that NSColor is not suported. The console fills with messages:

CGContextSetFillColorWithColor: invalid context
CGContextSetStrokeColorWithColor: invalid context

The reason behind this seems to be that cocoa attributed string uses NSColor, Core Text uses CGColor the two objects are not toll free bridged and are not the same. This does no look like a hard thing to fix up after reading, and before writing.

One of the advantages of using the Cocoa RTF import is that, unlike the Cello import, it supports unicode.

Here is a screenshot of this post saved as RTF being drawn by Core Text with a little unicode.

Converting Quartz Paths - MoveTo, LineTo, CurveTo

Cello has it's own support for curves - internally it has support for cubic and quadratic splines, as well as lines - the equivalent to what exists in CoreGraphics (Quartz) and generally in vector graphics systems. Within Cello there are classes that represent multi contoured splines and the attendant trickery to draw and manipulate them. They are of significance to me now as the existing Cello code requires the geometry of glyphs in its own internal format. So I needed to write conversion functions from a CGPaths to Cello's internal format.

Cello's curve format is unusual so writing the conversion has taken a little time. Most graphic systems that I have come across describe things in the form of, moveto, lineto and curveto. The idea being that "from where the pen is do this". Cello seems to be the other way round. Instead the curve verbs appear to be of the form linefrom, curvefrom, and movefrom.

My approach to writing the conversion has been first lift some existing code that converts the output of the D-Type font scalar (which is similar to the way that CoreGraphics thinks of curves) and to write a converter from Cello's internal format back to a CGPath. Again for this I could lift some code from Cello's output - again in the output the format paths are similar to CoreGraphics. Using the to and from converters it was possible to write Unit Tests to make sure that an original path is the same as a path converted, and then converted back. CFEqual will let you know if two paths are equal or not.

CPTAssert(::CFEqual(origPath, shape.AsPath()));

Converting and then back converting to, demonstrably, the same thing gives some idea of correctness but it does not guarantee that what you have done is correct, it just asserts that these two components are consistent. When I can look at the glyph shapes in the final output format I will be able to check this.

CoreText and Unit Testing

Working through the text the font handling in Cello I find that it meshes well with Core Text. The extent of what I am currently doing requires me to create CTFonts and get glyph curves. This is very similar to some early work I was doing in Cello and as I progress a few modest functions into my Core Text support there is an opportunity to share code.

If you have not used Core Text before it is a little different in terminology - the CTFont is a little more akin to a style than I am accustomed to and carries bit more information than say an ATSUI font (ATSUI will be deprecated in Snow Leopard). For example a CTFont has an idea of size and transformation matrix. Deploying this is not rocket science and it is more a question of jumping through the hoops.

I am unit testing the functions that I write as I add them - also I am taking he opportunity to fill in a few gaps in work that did earlier. For example I have written a simple test for the retrieval of kerning tables. To have any kind of stability unit testing can't depend on fonts on your machine - these can and do vary across system releases. Relying on fonts that are installed on a particular machine is just as fraught.

As a solution I am embedding the few fonts I use in unit test into the test bundle, as I have done with GIFs, PNGs and JPEGs in earlier tests as covered in an earlier post here. At the start of a given test I can activate them using ATSFontActivateFromFileReference, and deactivate them afterwards using ATSFontDeactivate. This means that I can write tests that depend on particular fonts.

Note ATSFontActivateFromFileReference will be deprecated in Snow Leopard - however there is no Core Text function currently available that will activate fonts.

Starting Text

One of the real pleasures of working on Cello is the quality of the code and commenting. I have started working on text. The two big things left on the file format are text and sounds. Text is, I think, the bigger - but is more familiar to me.The first step is to compile in the registration object and then to work through files that need to be built and included, to add and then build them. The text is a little different from other parts of Cello in that it is complex and has a lot that is dependent on the text engine (D-Type) as well as Windows. Ultimately all of the basics will need to be moved over to Core Text. There is a lot to understand. It is just so nice to open a header file that kicks off like this:

DWATextItemStyle holds style information that applies to a text item. Bit flags for each property determine whether that property is valid in the style. Invalid properties are used for leaving some properties 'as is' when setting the style, and when a property value is not consistent across the whole selection when getting the style.

Reading Bitmap Objects III

[build 0034] It looks like I have come to the end of DIBs and CGImages - well that is as far as the file format and file generation goes. The final part was, much as I suspected, mainly leg-work.

Below is one of the final parts of my testing - showing transparency working correctly in the output file. The GIF transparency has the characteristic halo as it only supports 0% or 100% transparency.Testing the file generation had a few wrinkles - the file format caters for 8-Bit indexed bitmaps, but I was only able to get the Image IO file import to yield 16bit indexed bitmaps. So in order to test this properly I added code that would convert the 16 bit indexed to 8-bit indexed. In the end the extra leg-work was well worth the effort - the unit tests that I added to check the conversion threw into open some underlying issues that I had in some of the other conversion routines.

Prior to moving on to the next part, I shall probably start looking at text and glyphs, I added a small number of sample files to the Unit Tests of the project - to assert that a file, with images, when written is the same as when it was first read (I create these test files in the PC version of Cello) - the idea is to get an early call on any regression in this area.

Saving JPEGs with Image IO

[build 0031] JPEGs now work again.

Fixing the problems in the JPEG images fell out in the wash. It looked like a byte ordering problem combined with a pixel size problem (which would explain the vertical striping) - but I did not look into it. Instead I used Image IO to generate the JPEG. The save, using Image IO, fell out in four or five lines of code. Using Image IO means that the OS will handle all the byte order, pixel order, packing etc - all I need to do is to feed it a CGImageRef. As all bitmaps in the OSX version of Cello are now CGImages this is natural. Also as a tiny fringe benefit using Image IO also means that I have been able to remove the JPEG library from the project build, so the file count and build time are modestly improved.

All that remains, to put the issue of bitmaps to rest, is to work through the remaining problems with the other bitmap formats that are not encoded as JPEG. This would seem to be more of a handle turning exercise - there is code missing for the 16-bit indexed bitmap and there may be byte/pixel ordering issues for the 32 bit. Byte/pixel ordering issues happen because the original code expects certain byte orders and I need to move this over to using the more general pixel unpacking code that now exists in the project.

DIBS and CGImage - The price of progress

[build 0030] Sometimes the price of progress is that things stop working. Not much works in Cello but the JPEG based part of the export now no longer works.

For the past couple of weeks I have been working out the underlying problems with bitmaps, DIBs and CGImages. I abandoned my approach of getting the melding the CGImage implementation to be more like the MFC Dibs and instead more-or-less changed the DIB class so that it was a thin wrapper around a CG image. This was what I originally intended to do but early on the path of lest resistance seemed to be else where.

I don't think getting the JPEG export to work properly will be a big deal and on from that the other graphics formats.

More Bitmaps - DIBS and CGImage

[build 0029] I have been working on bitmaps and bit images and the whole thing is slowly coming more into shape. The issue that I have been having with bitmaps is that fundamentally the bitmaps that are imported into Cello using Image IO as a CGImage are richer and more complex than those represented by DIBS. A CGImage has notions of byte order (big and little endian) and different data packing. Below is a classic example of the sort of an endian problems that you can get.Additionaly there are wrinkles such as GIF images and PNGs with a color table are imported as indexed 16 bit colors. One byte is an index into the color table the other is transparency. Also internally Cello wants to deal with non premultiplied alpha, where as the export needs the alpha pre multiplied.

In with all of this I have discovered that it is possible to get a direct copy of the pixel data of a CGImage - using CGImageGetDataProvider.

With all of this I have changed my approach a little. I have added a CGBitmapInfo to the OSX version of the MFC bitmap header so that I have an idea of byte order and packing. I have also written, and tested, some templated pixel accessors so I can easily access the various channel packing orders.

All of this needs to be integrated with the existing image export so there is still a fair way to go before the image side of things is wound up.

Reading Bitmap Objects II

[build 0027] Looking at the problem with the images it initially seemed like it might be a problem with bounds calculations. Initially I suspected my implementation of CRect intersection operation, that I had added to get the image import code to link, might be at fault. This was a hunch despite the fact that I had written Unit Tests for this essentially trivial function. Like many hunches it was quite wrong.

Later stepping through the code it occurred to me that I could have interchanged a hight and width when reading image dimensions. Again no problem - however I was not far wrong. As I stepped through the code where the DIB is set up from the CGImage I spotted this:
In a move that defies belief (rookie error) all the test images I worked with were 100px x 100px. Corrected results are a little better:

Reading Bitmap Objects I

Cello's file format is broken up into a number of distinct objects. The simplest the shape works and publishes correctly. All the object types are registered through a GUID and the read calls a factory to make objects that correspond to that GUID. This makes for an easy incremental development. All the registration *except* for the shape object is commented out. It is possible to uncomment them bit by bit and to work through them.

The plan is to work through the objects, register them, add the files that are required (working through compile and link errors) and then test.

Having registered the bitmap GUID and worked through the comparatively few errors and additional files required, and a crash in the Color Palette class bitmaps read and write. Publishing yielded a blank (missing) image. This is because the file format, under some circumstances" contains a reference to the image as a hard path. The fall-back behavior at publish time for a missing image is to just omit it.

Having a hard path to an image presents some problems for testing. the work around is to have a mechanism that will change the path in a file after the file has been been read. I did this by having a simple class that has a list of files with full paths and will match a file against the list based on the file name - so

/users/scratch/nonsense.jpg

will match against

C:\Documents and Settings\All Users\Sample Pictures\nonsense.jpg"

This kind of matching is easily done with Core Foundation's CFURL. CFURLCopyLastPathComponent will get the last path component of a path i.e. nonsense.jpg and URLs can be constructed form POSIX, HFS and Windows paths.

Having added the code - it is clear there are some problems with the publishing. See below:

Cello - publishing the first file

[build 0025] Cello now publishes the simple test file (just a red square) under OSX.
The publish revealed a single bug - code that was designed to ensure that all the folders in a file path existed, and created them if any were missing, failed. Cello has a class that does this and calls through to one of a small number of utility functions built on top of the cocoa NSFileManager. When writing the utility functions I wrote unit tests but importantly did not for the class that used them. It was left as a note on my todo list - and that was it. Writing the test and making the fix was a ten minute job. Had I done it when I identified the need I would have got a home run (or is it a maiden over) on the first publish, that is it would have worked first time.

Cello - opening the first file

[build 0024] Cello will now open a very simple file (just a red square) and save an identical one under OSX.

Getting Cello to open a file brought in a fair number of new files. Objects within the Cello file format are referenced by a unique identifier (actually a GUID) which are constructed from a factory. So adding the code to register the simplest object (a basic shape) with the factory had the effect of bringing in a reasonable number of new files - and once again working through the compile and link errors. The trick here was to be firm with the UI - unlike the underlying files the object files have UI references in them - so I added #if _NO_UI directives about these and their includes to stop any UI being sucked in.

Once I had done this (a couple of days ago) Cello opened her first file without crashing. The question then was "well so the read seemed to work but what did it read?" How can I have any certainty (beyond the fact that it did not crash) that everything was read - the file format is complex. My initial solution was to write the file out again and check that the file could be opened with he PC version of Cello.

Getting the write to work took time - there was a problem in the file class for files with a joint read/write access (there was no unit test for this). Having identified the problem I added the unit test and fixed the code by getting the new unit test to work properly.

When the write worked I then checked the files in HexEdit and they where different - it seems that Cello stores the file path of the file at the point of saving in the file. From here came an idea - adding a way that a chosen file-path can be saved will allow some files to be written that will be 100% the same as the original which in turn will allow unit tests for this to be added. Basically I can create tests that will read and write files (the original being a part of in the test bundle) and assert that the two are equal. I very much want to do this so that I find out as soon as possible if something I do breaks Cello.

Mounting the board - Cello's first crash.

Cello now crashes. This represents progress as it means that I have come to an end of rounding-up in the great ranch of link errors.


Cello won't do any more than allow you to choose a file - it crashes way before any part of the file reading code is called. So now to work through the crashes - I fully expect this process to pull in more and more files.

I expect that I will have to largely pause my process of Unit Testing - simply because there is nothing discrete to test. When I get to the stage of being able to open simple files I will try and add them to the Cello project as unit tests - the test being little more that the fact that Cello can open the file. The idea behind this is that if/when I break things so that a file that did open no longer opens I want to know.

CGImage & ImageIO - converting premultiplied alpha

The final part of the DIB (bitmap) equation is reading images. Cello has classes for reading and writing PNG, JPEG and a few formats including PaintShop Pro that are prevalent on windows. The change that I have made here is to use Image IO. Image IO will let you read all the most common formats - and also has support for RAW images (as spat by many digital cameras) amongst other things.

ImageIO is really easy to use you just need to give it something - a URL or a Data provider or whatever and it will turn it into a CGImage. The only issue with CGImage is that it is not possible to get to at the pixel data. In Cello there is a fair amount of code that manipulates pixels directly - so I have to facilitate this.

A solution is to draw the image onto a bitmap where you own the pixels - CGBitmapContextCreate will create a bitmap context and the following code fragment will draw it.

   CGRect drawRect = ::CGRectMake(0, 0, width, height);
   ::CGContextTranslateCTM(context.get(), 0.0, float(height));
   ::CGContextScaleCTM(context.get(), 1.0, -1.0 );
   ::CGContextDrawImage(context.get(), drawRect, theImage);

In this there are a couple of wrinkles.

• CGBitmapContextCreate won't work on an indexed bitmap. So all the images will be converted to either RGB32 or RGBA32. I don't see this as a big deal.
• CGBitmapContextCreate will only work with premultiplied alpha. Cello assumes the alpha is not premultiplied. The easiest thing to do is to wholesale convert the bitmap to nom-premultiplied alpha after the fact.

The code for this is reasonably straight-forward:

inline UInt32 Unpremultiply(UInt32 c, Float32 inverseAlpha)
{
UInt32 result = UInt32(Float32(c) * inverseAlpha);
result = std::min(result, 255);
return result;
}

void BitmapClass::UnpremultiplyRGBA()
{
UInt8* pRow = reinterpret_cast(m_pBits);
for (int row = m_height; row > 0; --row)
{
    UInt32* pPixel = reinterpret_cast(pRow);
    for (int column = m_width; column > 0; --column)
    {
        UInt32 alpha = GetAValue(*pPixel);
        if (alpha)
        {
            Float32 inverseAlpha = 255.0/alpha;

            UInt32 red = Unpremultiply(GetRValue(*pPixel), inverseAlpha);
            UInt32 green = Unpremultiply(GetGValue(*pPixel), inverseAlpha);
            UInt32 blue = Unpremultiply(GetBValue(*pPixel), inverseAlpha);

            *pPixel = RGBA(red, green, blue, alpha);
        }
        else
            *pPixel = 0;
        ++pPixel;
    }
    pRow += m_iByteWidth;
}
}

It is, of course, entirely non-optimized - but as this will happen as a single hit when an image is read I don't see it as being any kind of bottle neck. If it turns out to soak up time and that this is a problem I will attack it later.

XCode unit testing and Bundle Resources

In order to test the code that reads images I needed to set up tests that read images. In order to do this repeatably the best thing seemed to be to add them to the bundle of test application. In XCode this is quite simple - I just dragged in a folder of test images into the right place in my source files and XCode magically knows that these are to be copied - updating the copy files phase of the project.
Having done this the first part of my unit test was to get a url of a PNG in the bundle:

 CFURLRef png1 = ::CFBundleCopyResourceURL(
    CFBundleGetMainBundle(),
    CFSTR("DWCGImageSource_100x100a.png"), NULL, NULL);
 CPTAssert(png1);

This test failed immediately. It failed because URL of the main bundle when running the test is /Developer/Tools. Increasingly my calls to core foundation are lightly wrapped and CFBundleGetMainBundle() is a default parameter to my functions that get thr URLs of things relative to bundles. The solution I found was to add a #define _UNITTEST in the global defines and to write a new function to get the bundle.

 inline CFBundleRef GetMainBundle()
 {
 #if _UNITTEST
  return ::CFBundleGetBundleWithIdentifier(
    CFSTR("com.badbase.cello.unittest"));
 #else
  return CFBundleGetMainBundle();
 #endif
 }

I then use this GetMainBundle in the places where I used CFBundleGetMainBundle.

DIB to CGImage - the washing up

I changed the plans and in the end kept the class that manage the DIBs almost unchanged. The DIB class, within Cello, represents a bitmap with a physical bit-image. Give or take the fact that it can handle native compressed bit-images it handles the concept that an image is a bunch of bits and possibly a color table. A CGImage is very similar except that you can't get to the bit-image. You can make a CGImage where you can get to the bits if wrap the the bit image in a CFDataProvider. So the natural thing seemed to keep the DIB class more or less as it was but to provide a way off creating one from a CGImage.

The key change that I have made to the DIB class is to change the way that pixels are stored; I have moved from using a raw pointer for the pixels to boost::shared_array. The boost::shared_array is an array with an owner count - and it means that it is possible to deal with joint ownership. The point being that data provider needs to have ownership of the data it is providing (the pixels) as does the DIB class.

The other part of the equation is the colour palette its self. A colour palette is a list of colours - little more. In Cello there is a class that represents this list of colours as a list of RGBQUAD from which it can create the HPALETTE etc required for MFC bitmaps. I have changed the colour palette class so that it is now a thin wrapper around a CGColorSpace. You can still access a representation of it as a list of RGBQUAD - this is required for other things in Cello such as the ditherer. Changing this code and writing the attendant unit test was quite straight forward. I use boost::intrusive_ptr to manage the CFColorSpaceRef, and boost::shared_array to manage representation (cache) of RGBQUAD. In retrospect I think it would have been better not to have made this change and instead to have just provided a way of converting into a CGColorSpace. It would have been a smaller change.