Content Migration to a CCMS: Avoiding the Avoidable

JANA interview / Content migration

Content Migration to a CCMS:Avoiding the Avoidable

Insights from Rahel Bailie’s conversation with Stephon Johns on preparing content for a successful migration.

Rahel Anne Bailie

Written by Rahel Anne Bailie

This white paper draws on an interview that Rahel, founder of Content Seriously, conducted with Stephon Johns, Director at JANA, who has successfully completed a significant number of content migration projects, converting content from formats such as Word, FrameMaker, InDesign, PDF and even microfilm into structured XML, particularly into DITA. This paper sets out why organisations move content into a structured format for a Component Content Management System (CCMS), where migrations most often go wrong, and how to reduce the risk.

Watch the original conversation

Rahel Bailie with Stephon Johns, JANAWatch on YouTube

Migration problems start with source content

Most migration problems originate before conversion begins: in the condition of the source files and in decisions left unmade. Legacy content often carries years of inconsistent formatting, missing metadata, and editorial conventions that convey meaning to humans but not machines. Automation is essential at scale, but automated output still needs human review. Organisations that define their rules in advance, pilot with their hardest content, and keep people in the loop avoid most of the problems encountered during a content migration.

Why organisations convert content

Consider the documentation for a complex product. Such a manual can run to several thousand pages, and most of them describe variants and options that do not apply to the reader’s own product. When a problem arises, somewhere inside are the few sections that a reader needs. A service technician working against the clock faces the same challenge. In industries where billable time gets lost to fruitless searches, being able to locate information quickly is a competitive advantage.

Findability may be the first argument for conversion, and turning content into machine-readable components is the first step towards that. Some content exists only as PDF or paper. Some lives in software at end of life—for example, older versions of FrameMaker no longer receive bug fixes or security updates—and needs to be moved to a newer format. Some organisations also want content interoperability; content held in XML can move to another system if a vendor disappears, which saves a new conversion project should an organisation need to change vendor.

As well, regulation adds pressure to be able to recall content in various configurations. Right to Repair legislation brings a digital passport requirement with it. The passport calls for traceability and continuity, which means adding and retaining metadata so a conversion does not lose the history of the content. This is all part of a successful content migration.

Common problems when converting text

Source files are notorious for their lack of structure or, when there is structure, it is wrongly or inconsistently applied. There may be some basic structural elements, such as heading levels H1 or H2, but other styles are often applied based on how they make the text look on the page, and not for machine readability. Local overrides, such as using normal text and adding in-line formatting to make text look the same as a different element, are quite common. The result looks fine in print, but a machine cannot tell the paragraph from the heading. This can create problems once the content is moved, in bulk, into the CCMS.

PDF layout and table recognition

PDFs are particularly problematic. Many legacy PDFs record only how a page looks. A machine sees indentation, bold text, white space, and position on the page, and has to infer meaning during the conversion. A human can distinguish a heading 2 from a heading 3 at a glance; a script likely won’t be able to make that distinction. Tables used for alignment make matters worse, as tables are problematic to begin with.

Tables without borders, or without vertical borders, are not recognised as tables by scripts or AI engines. To spare clients from having to add borders to every document before conversion, the scripts must be trained to spot the borders.

Styles, lists and footnotes

Word and FrameMaker files are slightly easier to convert, but only when authors applied styles in a disciplined way. In that situation, conversion becomes a mapping exercise that most scripting languages can easily handle. However, that is a best case scenario. The usual scenario is the need to migrate a decade or more worth of work done by multiple authors, across multiple style changes. Styles such as Body, Body 2, Body Fix, and Body LeftAligned look identical on screen. These inconsistencies, multiplied by the dozen, have an impact on the conversion process.

Lists and paragraphs that break across pages or columns convert as two separate lists or two separate paragraphs. They will need human oversight to be stitched back together to their intended form.

Footnotes have their own peculiarities during conversion. Table footnotes and page footnotes behave differently and, in older documentation, neither exists as a real footnote. A footnote is just a superscript number and smaller text. Another common problem is that some authors used double asterisks or brackets, sometimes within the same document, which is hard for a script to interpret.

Common problems when converting images and metadata

Converting graphics is another common problem area. TIFF and EPS graphics are common formats used; modern browsers no longer support either format. The migration of the content needs to include conversion of the image formats, and remapping of all of the filenames, as well. Graphics with callout text cause a different mess. The text is extracted and dropped into the XML as stray words floating near the image.

Metadata is usually the largest problem. Many files have only the metadata built in by the software—for example, stylesheet headings in MS Word—and no enriched semantics to help the conversion scripts understand how to apply the needed metadata during the migration.

Links can also be problematic to convert. “Click here” cannot be understood by any script. A computer does not know what “here” is, and neither does AI. Web addresses and mailto links generally convert cleanly, but vague references need a decision from a human about where links should point.

Conversion automation and human review

Converting 500 pages (or 5,000 or 50,000) by hand is impractical, so every migration should use automated conversion to some degree. The decision is how much judgment to hand over for automated conversion, and how much needs human intervention.

Dropping documents into an AI tool with no rules will produce a converted file, just not necessarily with the intended result. The rules come first. The conversion plan must settle the metadata strategy, the topic types in use—if DITA, then concept, task, or reference—where topics split up, whether a chapter becomes one topic or several, how the book map is structured, and which attributes are needed. And the greater the deviation from the standard schema, the more intervention will be needed.

Consistency matters. A script converting a problematic document the wrong will recreate the same problem the same way every time. Twelve authors can produce twelve different error types.

Solid approach to content conversion

The recommended approach to content conversion is to start by asking whether the documents are allowed to be converted automatically using AI at all. In some organisations, this approach is not allowed. In other organisations, they run models inside their own firewall.

Deterministic work can go to scripting languages such as Python, with AI reserved to process edge cases. AI is good for spotting reuse candidates, inferring that two sentences have the same meaning, and catching editorial inconsistencies. AI also does well at the delivery end. Ask for the steps to service a specific model of a product, and the system can retrieve the right steps, with the documentation behind them. This is kinetic information: content delivered to the reader instead of the reader having to search for it.

The primary risk is whether the output can be trusted. XML can be valid while the technical content is still wrong, and bad documentation in means bad documentation out. An AI model may decide that two documents are identical when they are not. Similar inconsistency appears when the same task runs through two different models, with suggest markup varying from model to model. Guardrails need careful settings, and the governing rule is human in the loop: a knowledgeable human needs to check all output before it gets migrated to the new system.

XML, Markdown and JSON

Is migration to Markdown or JSON practical? Writers regularly ask for Markdown, and digital agencies regularly ask for JSON, with the rationale that this is a format used by AI models. However, LLMs perform better on XML for inputting data because of how the tags enclose information. Leading AI research labs, particularly Anthropic, officially recommend XML. Markdown covers headings, paragraphs and some table formatting, and that is about all. It has no construct for metadata. JSON handles properties, as names and values, and suits programmers. Documentation with rich metadata and reuse needs more structure than either Markdown or JSON provides.

Start with the hardest content

Manual conversion may be acceptable when there is no hard deadline. A migration tied to a wider digital transformation is a different matter. The most common mistake when launching a content migration project is poor preparation. No off-the-shelf tool converts every source to every target, and without the right expertise, attempts at conversion become three steps forward and one step back. The team should define its source material, its variations. and its target output, then create the plan.

Having the right expertise, as mentioned earlier, is important. The project needs an information architect who understands the target output, can do a style audit, has an understanding of graphics, and set up a QA process. Few organisations hold all of that expertise in-house, and the mix of skills changes with the project. Word to XML calls for one set of skills, FrameMaker into a CCMS calls for another, and every CCMS differs.

Three phases to a successful migration

The “crawl, walk, run” method makes the most sense, as long as it is undertaken strategic. The usual advice is to convert one small document first, but that trains on the easy cases and defers the hard ones. A representative sample works better: a medium-sized document with the messiest source formats, the worst graphics, and the most complicated configurations. If that conversion works, the rest is simple, and it produces every mapping the project will need. Then, a larger batch becomes a test case where the remaining kinks can be worked out. The final step is to set up the production line with the rest of the content to be converted.

A successful migration is decided largely before conversion starts. Organisations should audit their source files for style consistency, graphics formats, metadata richness, and the defect patterns described above. They should settle on a metadata strategy, topic types, and splitting rules in advance, and pilot with the hardest content. Deterministic scripts should handle repeatable work, AI should be used selectively and with permission, and a person should review everything before release. An experienced information architect can fill the gaps of most in-house teams. Content migrated this way gives an organisation structured content that smoothly migrates into the new system and can move between systems as needed.

Sign-up for the latest Bluestream News

Close