As he stood staring at them, they asked him no questions, for his face told them everything.

'I cannot find it,' said he, 'and I must have it. Where is it?'

His head and throat were bare, and, as he spoke with a helpless look straying all around, he took his coat off, and let it drop on the floor.

'Where is my bench? I've been looking everywhere for my bench, and I can't find it. What have they done with my work? Time presses: I must finish those shoes.'

They looked at one another, and their hearts died within them.

Dickens, a careful student of human nature, provides here a vivid portrait of Dr. Alexandre Manette, who, after being held 18 years in the Bastille, is released, but is unable to adjust to his new freedom, and in times of stress lapses back to the familiarity of his prison labors, making shoes.

We all have been prisoners of Microsoft Office and their proprietary file formats. You may no longer recognize it as a prison, because this cell has been your home for the past ten years, but here is what it looks like:

1. Editing a document requires Microsoft Office.
2. Since Office runs only on Windows, you also require Windows
3. These restrictions lead to a purely heavy-client view of document processing.
4. This also leads to a model of programmability that emphasizes storing executable code (macros/script) inside of the document, resulting in years of security nightmares. Here is a typical recital of the known dangers.
5. If you didn't want to put script inside your document, you could get to the data via Office automation API's, but this again required a machine running Windows and Office.
6. It also emphasizes a unilateral view of WYSIWYG which emphasizes early formatting and layout decisions and de-emphasized semantic richness in documents. For example, "What has WYSIWYG Done to Us?".
7. The tools that were created for us to record our thoughts with now instead constrain or even substitute for our thoughts. For example, PowerPoint Panders to our Weaker Points" in the Guardian, and Tufte's "PowerPoint is Evil".
   
8. The above also lead to a stifled market for 3rd party document processing tools. We will never see the value of what was never allowed to occur, but the opportunity cost of the innovation that did not happen in this single-vendor world is enormous.
   
9. This also lead to general lack of competition in the productivity editor market, leading to a decade of buggy products with little innovation.
10. We've been locked into a one-size-fits-all offerings of bloated applications. Many people are over-served by Office and therefor are over-paying for functionality they do not need, while others are under-served by the resulting products they cannot afford.
11. Functionality has been arbitrarily segregated into three and only three application classes, "Spreadsheet", "Word Processor" and "Presentation Graphics".
    

The move from proprietary binary formats to new standard formats, like OpenDocument, is a movement from imprisonment to freedom. Although the technical constraints have been lifted, have we really made the mental adjustments necessary to engage our new freedom? Or are we still silently pacing a 10-foot cell in our minds? If we merely recreate our cell walls in XML, then we are still prisoners.

I've been as much a prisoner as you have, and am a creature of habit, so don't look to me for all the answers, but I do have a few thoughts on what this new freedom will look like.

Instead of being opaque black boxes that can only be used on one vendor's system, documents will be transparent. Anyone can access them using whatever operating system and whatever tools they want, and for any purpose they want. Python on Linux, REXX on AS/400, and C# on Windows will all have equal opportunity.

This also implies that document processing will no longer be restricted, technically or by licence, to the desktop. Innovative things will occur on servers. We're starting to see some of that with Writely and wikiCalc. But that is only the beginning. We will see search engines that can intelligently search content for specific MathML expressions, spiders that will collect and aggregate slides from presentations and allow you to share them, document repositories that will automatically check citations in papers and calculate the intellectual social networks these imply, stock brokers that will allow you to download your statements formatted in a spreadsheet, with additional analytics calculated via spreadsheet formulas. Creating, editing, reading, viewing, storing, collaborating will be able to be done anywhere, from your cellphone to the largest servers.

Since the server typically has access to not only your documents, but your organization's as well, as well as easy access to other information about the users, such as your role and group via LDAP, an application can drive workflows that relate the contents of the document to similar content, as well as to you organizational role, and to your business. The companies that unlock the knowledge stored by your knowledge workers in their documents will be leading us into the next decade.

The old walls will fall that once segregated functionality into the arbitrarily prescribed boundaries of "Spreadsheet", "Word processor", and "Presentation graphics". Dan Bricklin is leading the way with his wikiCalc. Is it a Spreadsheet or is it a Wiki? If you have to ask the question then you are still a prisoner. The point is wikiCalc is whatever Dan Bricklin wants it to be. That is freedom to innovate. We will see the arbitrary divisions between application genres become fuzzy and fall away as we all recognize our new freedom.

Document programmability will be turned inside-out. Instead of putting code inside of the document, turning documents into virus vectors, the code will be carefully segregated. Once the code and the data are distinct, we can put the code on the server, where it can be more easily managed, maintained, and secured. This clean separation of code and data will be as important to system stability and security as was protected-mode in the 8026 processor which first enforced this data/code separation at operating system level. I see macro viruses becoming a thing of the past, like smallpox, because the importance of data/code separation will finally be enforced, and users will not be emailing around code disguised in documents.

We will start thinking of documents as data, and as inputs to modules that process data. I see visual design tools that will allow you to drag and drop a document template onto a design surface and expose various fields in the document which can be wired up to databases, web services or other data sources.

I see financial analysts creating financial models in spreadsheets, then converting the spreadsheet into a web app that can then be deployed anywhere to provide browser-based access and execution of the model via any browser.

I see a variety of productivity editors available at a variety of price points, from free, open source ones, to commercial offerings for desktop and other devices, to specialized offerings with extra features for vertical markets, like legal, medical, academic, or scientific uses.

I see an escape from documents-as-pictures, where users sweat over pixel perfection and pray that the applications don't screw them up. Today the end user doesn't worry about font kerning. We rely on the font managers to get this right, and we accept the results, and concentrate on what we, the authors, add to the document. We are freed from that mental burden of kerning. But why stop there? With smarter applications, we will be freed of most or all formatting burdens. We will concentrate on writing, not on styling, and rely on the applications to get the appearance right. This will free our time to give an increased emphasis on semantic richness, putting our knowledge and experience and outlooks and opinions into the document, and encoding it in an way that allows new modes of collaboration and redefines what a document is.

That is a gimpse at what freedom looks like to me. But let's not forget that being freed is not the same as being free. There are those out there who are attempting to merely recreate the same single-vendor closed system we've had for the past 10 years, and redoing it in XML. This may be a comfortable choice to those who have known no other way. But is it really freedom? I look out and see the jailer offering to sell 10-foot apartments to those just released from their 10-foot prison cells. Will you follow?

I mentioned that in my opinion, Sun were completely aware that ODF wasn't sufficiently defined to support spreadsheet interoperability as long ago as February 2005 and that the realpolitik inside OASIS was to take advantage of the EU IDA's request to standardise by rushing to be first despite knowing the ODF specification was deficient in at least this area.

• The lack of a spreadsheet formula definition in a file format documentation is unusual, defective and prevents intereroperability
• Spreadsheet formulas were left out because ODF standardization was rushed, for political reasons

• The OOXML Working Draft 1.3 is currently 4,081 pages long. At the SC34 meeting in June we were told by the Ecma Secretary General that more material was coming and that this draft was only 2/3 complete. By my calculations, this gives a final size estimate of around 6,000 pages.
• Predicting the completion date is harder. But we do know that Ecma specifications can only be approved twice a year at Ecma General Assembly which are in June and December. If I were Microsoft I'd really really really want OOXML approved in time for the Office 2007 launch, so I'm predicting Ecma approval will be sought at the December Ecma General Assembly.

Of course I could be wrong on either or both of those estimates, but let's see where the logic takes us. The following table summarizes the time under standardization as well as the rate of standardization (pages/day) for each specification.

Amsterdam - August 14 2006 - OneStat.com, the number one provider of real-time intelligence web analytics, today reported that Microsoft's Windows dominates the operating system market with a global usage share of 96.97 percent. The leading operating system on the web is Microsoft's Windows XP with a global usage share of 86.80 percent. Microsoft's Windows 2000 has a global usage share of 6.09 percent and is the second most popular OS on the web.

The global usage share of Apple's Macintosh is 2.47 percent and the global usage share of Linux is 0.36 percent.

• Confidence level = 95%
• Confidence interval 0.005
• Population: 1000000000

With 95% confidence Linux has a global usage share of 21.1% (+/- 0.1%) and Firefox as a world wide usage share of 46.5% (+/- 0.1%)

I. OpenOffice.org Conference (OOOoCon 2006) in comming up, September 11-13th in Lyon, France. The last day starts with a panel discussion of ODF topics, and follows with a track dedicated to ODF. I'm on at 14:00 with a presentation with the exciting title, "A Technical Comparison: ISO/IEC 26300 vs Microsoft Office Open XML (Ecma International TC45 OOXML WD 1.3)".

The abstract is:

Two XML office file formats have been pressing upon our attention, the OASIS OpenDocument Format, recently standardized by ISO, and the Draft Ecma Office Open XML. This presentation will review history of each, the process that created them, and examine each format to compare and contrast how they deal with issues such extensibility, modularization, expressivity, performance, reuse of standards, programability, ease of use, and application/OS neutrality.

II. KDE enthusiasts get together two weeks later, in Dublin, for their aKademy 2006. Tuesday the 26th will be OpenDocument Day. I'll be there, and will give a lighting talk on something, probably related to some ODF programmability API ideas I've been having.

III. If you didn't see it yet, Rick "Schemetron" Jelliffe has an interesting post over at O'Reilly looking at ODF and OOML documents from the perspective of XML complexity metrics. This is a topic which Rick has done a good deal of work with in the past, so it is interesting to hear what he has to say. Did I see something there about OpenOffice loading documents faster than Office?

IV. The ODF Formula Subcommittee has set up a wiki page on our work defining OpenFormula. A lot of good information is there. This page will be updated with the latest status, so you'll want to make it the first place to go for the latest info on our progress.

...the standard [Phipps here speaking generically and not about any specific standard] should have been created transparently. Just as an open source community looks with concern on a large, monolithic code contribution, so we should be wary of standards created without the opportunity for everyone to participate or, failing that, with a full explanation of every decision that was made in its construction. Without that there's a chance that it's designed to mesh with some facility or product that will be used to remove our freedom later.

7. SCOPE OF LICENSE. ...You may not disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval

It's certainly going too far to call Microsoft's currently lukewarm support for ODF a change of heart, but it is nonetheless an encouraging instance of Microsoft listening to its customers. We call on Microsoft to make way for ODF in the standard supported-file-formats list giving it at least the same stature as the formats of the suite's once-fierce rivals.

In that spirit I'd like to propose some evaluation criteria for what first-class file format integration looks like, so you'll recognize it when you see it. I'll use Word and ODF word processorr documents (.odt file extensions) as an example, but the points apply equally to other formats and other editors:

1. Is the format support "in the box", installed and configured when you do a default install?
2. When you open the default File/Open dialog (the one you get with the Control-O shortcut, the one we all know and have been using for 20 years), do ODF files show up? In other words, does the default file mask include *.odt, or can you at least easily select ODF from the drop-down list of file formats?
3. If you create a new document in Word, and then type Control-S to save the document, is ODF available on the list of file formats you can save to?
4. Can an end user or administrator (no coding required) make ODF be the default format for saving documents?
5. If you open an ODF document, and change it and then Control-S to save it, will this occur?Or will it save to some other format? Or will you get an error?
6. Is support integrated into the Windows shell via the registry, so you can double-click on an ODF file on the desktop, in a folder or as an email attachment and it will automatically launch in Word? If you are like me, 90% of the time you are not opening a document from within Word, but you are launching Word with a file in this manner

Is there anything here I'm missing?

Once again, the open source community wins. Not only is the fidelity of OpenOffice's support for MS Office formats higher, but they provide much closer integration of these competing formats.

I'm reading many explanations and excuses for why Office's support for ODF is the way it is. Here is a selection to ponder:

Brian Jones, a Program Manager for MS Office, said of the Add-in support:

It's directly exposed in the UI. We're even going to make it really easy to initially discover the download. We already need to do this for XPS and PDF, so we'll also do it for ODF. There will be a menu item directly on the file menu that takes to you a site where you can download different interoperability formats (like PDF, XPS, and now ODF).

I guess we differ how we define "directly exposed" and "really easy". IMHO, the easiest way to make this easy to find would be to install it with Office and put it in the normal place in the menus.

Also, in Office 2007 Beta 2, which I have in front of me, PDF and XPS are in the normal File Save and File Save As menus, not treated separately. I understand that there is a dispute with Adobe and so you are deciding to move the PDF and the competing XPS options out of the main File Save dialog, but that just proves my point. This is clearly a policy decision not a technical restriction. The fact is that PDF and XPS both work fine where they are in the File Save menu in Office 2007 beta 2.

Brian Jones again:

We already have the PDF and XPS support for Office 2007 users that unfortunately had to be separated out of the product and instead offered as a free download.

I agree that this is unfortunate. Full integration should mean it is included in the product and supported in the usual places. I'll take your world that PDF had to be taken out for non-technical reasons. But what prevents ODF from being integrated at that level?

And Jones on Interoperability by Design:

You see a lot of folks talk about interoperability, but often they just don't mean the same thing. From our perspective it's something we want to build directly into the products so that it just works.

That is a reasonable goal. I like it. By this definition, will we ever see a Microsoft-sponsored effort, open source or otherwise, to make ODF so it it is built "directly into the product so that it just works"? As pointed out above, a bunch of open source developers have already given this level of support to Office formats in OpenOffice. Isn't it embarrassing to be unable to accomplish the same task in Office 2007?

Patrick Schmid is a Microsoft Certified Systems Engineer who has obvious deep knowledge of Office Add-ins and customizations. He writes:

It is possible to write a custom file import/export converter for Word. This would add the file type supported by the converter to the file type lists in the Open and Save As dialogs. Unfortunately, this is not an option for the ODF translator project.

Further:

The Open & Save As dialogs cannot be customized: There simply is no way for an add-in to add another file format to the file type list in those dialogs. I checked this with the Office beta team just to be sure.

There is no way to do this? When the Open Document Foundation responded to the Commonwealth of Massachusetts's RFI requesting info about the feasibility of an ODF Plugin, their response included screen shots of what appears to be the File Open/File Save dialogs in MS Office allowing the read and write of ODF files, side by side with the built-in formats like DOC. I hope this will not prove to be another embarrassment, where open source developers accomplish another task which Microsoft says is impossible.

Schmid further writes:

Why should MS waste resources on providing one particular feature for add-in developers that most Office add-in developers (all not affiliated with Microsoft) would put at the very bottom of their wish list?

Keep in mind that a single ODF Add-in could satisfy millions of customers. So we shouldn't count the value by the number of add-ins which could be written, but by the number of end-users who would use the add-in.

Stay tuned. I don't think we've heard the last of this issue. In particular I'd like to hear more about this Open Document Foundation Plugin, and the closer integration they illustrated in their RFI response.

• I had misconfigured some Blogger settings, causing my feed URL to be incorrect. So, if you subscribed before 7/20 then you'll want to re-subscribe.
• I'm auto-converting Blogger's default Atom 0.3 feed to Atom 1.0. Thanks to the Sultan of Syndication, Sam Ruby, for help with that.
• I've added a brief capsule bio linked to from the sidebar. My contact info is there as well.
• I've also added a blogroll of sites which I find indispensable for keeping up with news related to ODF. I hope you find them as informative as I do.

I plan on writing additional essays related to ISO ODF and the draft Ecma OOXML. Expect to see one or two a week. There are already several bloggers who cover the universe of ODF news/happenings. Rather than repeat their good work, I will try to deal with technical issues in depth, with an emphasis on clearing the air of some of the FUD out there. I'll also try to keep it fun. If there are any topics you especially would like to hear, please let me know.

Now for fidelity. Since you may not all have Office 2007 beta 2 installed, I'm going to show you the fidelity via PDF exports. In all cases I manually verified that the PDF output was identical to what I saw on the screen, every error is real, nothing introduced by the PDF export process.

First up is a document I call "the sampler". It has a little bit of all the basic word processor formatting, fonts, alignment, nested tables, graphics, other character sets, headers/footers, images, captions, etc. It is not intended to be a particularly hard test of document conversion, but a basic test of core functionality.

So, here is the sampler, in the original ODF format, as well as the PDF rendering of it in OpenOffice 2.0.3, where it was originally created.

I then exported that file from OpenOffice to Word format. This demonstrates the quality of conversion users already get when running OpenOffice. Here is is in DOC and PDF exported after loaded the DOC file in Word 2007 beta 2.

Good, but not perfect. Some differences:

• the bullet point size larger in Word than in OpenOffice
• the nested table collapsed into main table in Word
• the above table problem causes the table to take up more vertical space, pushing the graphic onto a second page

Again, that is the OpenOffice --> Word conversion we all have available for free today in open source code. Since DOC is a proprietary binary format with inadequate publicly-available documentation, this level of fidelity is impressive. So moving from ISO ODF to Draft Office Open XML should be that much easier, especially since the target format is voluminously documented (4,000 pages and growing), and the writers of the translator are receiving technical assistance from Microsoft.

Let's take a look. From within Word 2007 (beta 2) I use the ODF Add-in to load the sampler ODF file, and get something that looks like this PDF.

I won't characterize it but to say it fared less well than I expected. Problems include:

• headers/footers dropped (data loss)
• bullet list indentation ignored
• number list indentation ignored
• table dimensions messed up
• caption for the graphics sized and positioned incorrectly

Whether these are all bugs or merely functional limitations is an interesting question. There is a Functional Specification document available on SourceForge for the Add-in which lists these requirement:

2.1.1.1. Basic Formatting

Here is the list of formatting items that the Add-in and command line translator would keep intact. The first 10 in the list are must haves and the last 4 (number 11 to 14) are good to have items of formatting.

1. Bold
2. Italics
3. Underline
4. Bulleting
5. Numbering
6. Indentation
7. Alignment (Left, Center, Right)
8. Font size
9. Font face
10. Tabs
11. Tables
12. Font color
13. Highlights
14. Background colors
    

Tables are "nice to have"? I'd hope so! This does not give me the impression that full fidelity is in their plans. Forget about scripts and macros. They are not even planning on tables or font colors. I hope I am wrong or misinterpreting their plans here, but that is the requirements document they have posted.