Database Publishing with FrameMaker Why database publishing Database Publishing

Database Publishing with FrameMaker: Why database publishing? Database Publishing with FrameMaker Why database publishing? Traditionally, databases have been used to manage vast amounts of data when quick access and flexible selection and sorting are crucial. Since a database’s primary mission is to store rapidly changing data, report generation tools have often stressed speed of report development over the presentation of information. Why spend time preparing a polished report when the underlying data will change by the time the report is finished? Publishing systems, on the other hand, typically focus on more stable information such as user documentation, manuals and instructions. As a result, most publishing systems offer tools to effectively communicate fixed information rather than tools that dynamically accommodate changing data. Technology has moved to the point where compromising between a database’s need for speed, and a publishing system’s emphasis on quality of output, is no longer acceptable. The technique known as database publishing bridges the gap between speed of report generation and documentation quality. At the 1997 Adobe Premium VAR conference, speakers talked about Adobe’s view of database publishing. In response to questions from the VAR audience, John Brandon, the former Vice President North American Sales, described the coming conflict between database engine providers and the desktop publishing software providers. He stated that both are trying to develop solutions in each others territory — database engine providers developing 1 1998 FrameUsers Conference: Using FrameMaker in the Real World more robust report writers and formatting engines and desktop publishing software providers developing database engine or connectivity. He also stated that he wasn’t sure who was going to win. This paper provides an overview of the emerging database publishing technology. It examines the real-world needs and uses that have spurred this development, and explains how FML has used Adobe tools to incorporate support for database publishing into their services and products. Benefits of Database Publishing The benefits of this technology impact any business that needs to communicate changing information internally or externally. Both information providers and users discover they are able to add new value to their existing information investments. Benefits for Information Providers ➤ Shorter production cycles for frequently updated documents. By eliminating much of the manual effort required to produce high-impact reports, database publishing reduces the time and cost of document production. For example, companies don’t have to compromise between the lost revenue from out-of-date price lists and the cost of manually producing updates. ➤ Opportunities for new types of documents and information distribution. Database publishing makes sharing database information practical. Where traditional methods were too slow — information was obsolete before it was printed — or too expensive, database publishing opens up new avenues for presenting information. Benefits for Users ➤ Improved quality of information. Users no longer have to trade timeliness of information for quality of format. The old standard text-based, columnar database report can be dramatically improved with professional quality publishing techniques. Database publishing offers readability improvements taken for granted in publishing but usually unavailable from standard database report writers. These include automatic creation of: ➢ Cross-references ➢ Footnotes ➢ Tables of contents ➢ Mixed text and graphics ➢ Hypertext navigation ➢ Indices ➢ Powerful table formatting Leveraging Existing Technology One of the key advantages of database publishing is that it allows businesses to leverage existing data and data management assets. Many of today’s database applications can create files usable by electronic publishing software such as Adobe’s FrameMaker and FrameViewer connected by a database-to-Frame application. By establishing communication between the two types of applications, the database can continue managing data, and the publishing system can be used as an information synthesis tool to gather data from a variety of sources (databases, graphics, and text) and present it in a single, cohesive document. This leveraging of existing tools provides several benefits: ➤ Provides flexibility to combine data from multiple sources. Publishing systems such as FrameMaker can synthesize data from different databases, e.g. financial data from one database, graphics and images from another, and text from existing documentation. ➤ Modular technology approach allows flexibility to use new and better applications. By combining separate systems into a modular database publishing solution, future enhancements to any individual component (a new object-oriented database for instance) can be implemented as part of the entire database publishing solution. ➤ Requires little additional training. Skills and experience to manage databases and publishing systems are often already available within businesses. 2 Database Publishing with FrameMaker: Database publishing requirements and products ➤ Improved on-line access to information. Database publishing provides opportunities for high impact presentation of information on-demand and in electronic formats or printed paper. Information customers can receive the information in the electronic and media form they need (PDF, HTML, SGML, paper, CD-ROM, WWW, etc.). Database publishing requirements and products There are three keys elements in a database publishing system: ➤ Database ➤ Text, table, and image formatting engine ➤ Inter-application program communication and control For complex requirements, this control application could even be an artificial intelligence-based system that requests different database reports based on the user's knowledge or security levels. The complexity or simplicity of this control application is defined by the user’s needs. Is an product suitable? While database publishing can improve the quality of output from many database systems, it may not be suitable for every publishing product. There are two general areas to consider when evaluating whether or not database publishing is appropriate for an application. Data Structure Requirements ➤ Is the data suitable for storage in a database? Database publishing is most effective when there is structured recurring information that can be categorized into database records that share common fields. Customer information such as names, addresses, and phone numbers are typical. Information that does not fit into a consistent structure works less effectively. Most information currently stored in a database can be used as part of a database publishing application. ➤ Does the data contain a mixture of text and images? While many databases can store image information such as drawings, illustrations photographic images, and video, many report writers are oriented toward text-only output. If your data contains a mixture of text and images, database publishing offers improved capabilities to manage output of the data. Output Requirements ➤ What are the output quality requirements? For many database applications, simple tabular listings of raw data are satisfactory. For other reports, database publishing can provide the necessary added value. The database provides a system for data entry, manipulation, selection and sorting. Some desktop publishing applications can integrate with virtually any database system. Databases that have previously been integrated with FrameMaker in a database publishing environment include Informix, Oracle, Sybase, Wacom, SQL Server, Access, and ACIUS 4th Dimension. One method of operation is for the text engines to read the database structured query language (SQL) output and dynamically manage formatting such as line breaks, page breaks, images, and header and footer information. Desktop publishing applications can also handle document management issues, such as importing referenced graphics, printing, viewing, saving and hypertext navigation of on-line database publishing output. Some database publishing applications are controlled entirely from the database system. The user runs the entire process from one application which controls the flow of information between applications. More complex applications may require an external control program to manage the interface between the database and the desktop publishing applications. In many cases this is a simple C program that issues SQL inquiries and controls which desktop publishing application template to use. 3 1998 FrameUsers Conference: Using FrameMaker in the Real World Reports that are similar to completed forms are usually good database publishing candidates. These might include mail-merge applications, fill-in-the-blank contracts, invoices, purchase orders and other form-like reports. Since such forms often require a mixture of text and images, they are often easier to produce with a database publishing solution than with standard database report writers. ➤ Is the typical output a limited set of standard reports or mostly one-time, ad-hoc reports? While database publishing can help improve the quality of one-time, ad-hoc reports, the real cost and timesaving benefits occur when there is a limited set of predefined formats that are used over and over again. ➤ Are indexes, cross-references, and table of contents important? Does the output combine several reports into a single longer document? These are all structured document issues ideally suited to electronic publishing software such as FrameMaker and FrameViewer. ➤ Is it necessary to distribute the database information in a variety of formats? FrameMaker has direct, easy to manage pathways to: ➢ Adobe’s PDF ➢ HTML ➢ SGML ➢ Printed paper ➢ CD-ROM ➢ On-line FrameMaker can also be used as an effective input source for databases. The FDK can be used to develop an input system that will allow author/editors the capability of entering information once and have both a FrameMaker document and a database updated at the same time. The diagram on the next page shows the possible information flow paths using FrameMaker as the formatting and conversion engine for database publishing. What publishing products are candidates? Database publishing can be used to effectively produce superior: ➤ Catalogs ➤ Directories ➤ Manuals ➤ Reference books ➤ Reports ➤ SGML documents ➤ Any pattern-based published documents Catalogs Catalogs may represent the largest database publishing market opportunity. All manufacturers and distributors publish catalogs for distribution to their customers. Directories The directory market segment is comprised of at least two major product components, 1) books and 2) phone directories. Books. Directory books are published by a variety of organizations. Associations publish directories for their membership. Speciality publishers publish information lists for their customers. Business publishers publish corporation directories to provide contact and/or investment information. Phone Directories. Phone directories are published by a number of different organizations including the regional Bell operating companies (RBOC) and subsidiaries, yellow page speciality firms, and other third party organizations. Any given area of the U.S. may have five or more directories published by competing firms. Manuals The manual category includes parts, service, and test manuals. The size of this database publishing market is not clear because the value of this market is hidden by businesses in the cost of getting a product to market. Typically 4 Database Publishing with FrameMaker: Database publishing requirements and products manual development and maintenance costs are not delineated as a line item in a company’s performance. This database publishing market is probably small, in spite of the fact that there are companies that specialize in the market. Reference Books The reference book market is another small database publishing market. Most likely less than the previously mentioned markets, it probably involves more support services than any other market because of reference book’s uniqueness. Reports The reports category represents a very large number of types of database publishing documents. Reports may be financial, statistical, and/or textual. The largest report publisher is the U.S. government which publishes thousands of documents that fit this description. SGML Documents Database publishing techniques can be used to publish SGML documents, too. The width of this market is unknown at this time. Other Pattern-based Documents Its not clear what other pattern-based documents have not been included. This category is reserved for all other documents that don’t fit within the previous descriptions. Database publishing systems have the ability to generate output in PDF directly from a database using FrameMaker in batch mode operation. Because of this feature and because the applications can be controlled by other processes and database information, they can be used to perform WWW-based interface directed database publishing. This makes it easy to generate PDF dynamically from a database using a WWW front-end or as part of some other batch mode process. Dynamic PDF documents can be created using a WWW interface to the database publishing applications and the publishing database. Using Java, CGI, or other WWW programming environments, simple applications can be written which allow: ➤ mission specific information to be displayed on an HTML or XML page ➤ a user with a WWW browser to choose information of interest ➤ population of the publishing database with the user's selection ➤ initiation of database publishing using the application and creation of a user ➤ defined PDF document ➤ email transmission of the PDF document to the user This process allows users to create documents from a database(s) and have them delivered using a WWW interface or email attachment — extending the reach and scope of database publishing to make it user defined and accessible to the world through the WWW. Practical applications include custom: Training course catalogs Educational and corporate organizations can list their available training courses on a WWW server for access. Training customers can select the courses they are interested in and have a PDF listing of the courses, including title, number, time, location, description, prerequisites, and other pertinent course attributes, emailed to them within minutes. WWW Interface Database Publishing Another emerging database publishing arena is the WWW interface driven database publishing system. Although the system still produces the same publishing products as listed above, the system differs in that the end customer usually specifies their own product content and its performed on the Internet. In this environment, a WWW page is used to prescribe the document options to an information customer. Once customer makes a choice, a document is processed from a database and transmitted back. 5 1998 FrameUsers Conference: Using FrameMaker in the Real World Product catalogs Manufacturers can list products and catalog attributes on a WWW server enabling distribution centers to create customer specific catalogs. A salesperson in a remote location can go to a protected WWW site, access the appropriate page and application, choose the appropriate products and format (including end customer specific heading and footers), and create a PDF catalog. Once the PDF catalog is received and reviewed, the salesperson can 1) print the catalog on a desktop printer, 2) go to a quickprint shop to print a number of catalogs, 3) go to a high end printer for large quantities of catalogs, 4) provide the end customer with a PDF file. Reports Numerous organizations develop databases for reports for use by internal customers, clients, or the public. Using the WWW and a publishing application, systems can be rapidly developed and deployed which will allow WWW enabled customers to create custom PDF reports from various databases. ➤ Associations ➤ Distributors ➤ Financial ➤ Government ➤ Health Care ➤ Manufacturers ➤ Printers ➤ Publishers The largest single market sector is most likely the manufacturing market since several products, including catalogs, directories, manuals, and reports are published there. The second largest is probably the government sector, including federal, state, and local. Unfortunately, the DoC standard industry classification codes are not broken down in a manner that allow segregation into these market sectors. FML is continuing a document search to determine these values. What publishing organizations are candidates Organizations that currently have shown a need to perform database publishing tasks include: A brief history Database publishing appears to have originated in the main frame computer environment. There are some indications that this may have occurred as early as the 1960s when there was a need to get information from large databases onto paper for personnel to read and analyze the information. Database publishing systems were awkward and written in COBOL for specific information functions. It is somewhat difficult to distinguish between early report writers and publishing because of the lack of capability for print formatting. Desktop database publishing did not start with the introduction of the desktop computer and word processor. Database publishing was too complex of a problem for the early software. Some of the first desktop solutions became available in the early 1990s. This includes: ➤ db Publisher (1992) ➤ Brio Publish (1993) However, there is no documentation to precisely describe these applications’ market history. We are aware of other proprietary applications, but once again, historical documentation is not available. 6 Database Publishing with FrameMaker: The many ways The many ways There are two basic methods to develop database publishing systems. One way is to create systems from scratch. The other way is to use available software and develop applications and communication code as necessary to achieve the desired results. Initially, all database publishing solutions were created by programming. As the desktop publishing market developed, more applications became available for integration to provide solutions. Currently, most database publishing systems are created by integrating applications together. There are few top-tobottom systems being used and sold in today’s market. However, these solutions appear to be decreasing in number of installations while the integrated solutions appear to be gaining market share. Looking at the integrated database publishing systems, it appears that solutions using Quark’s Xpress have the largest market share. Although no specific market publications track this attribute, from observation of installed product, number of products, and number of licenses believed to be installed (Xpress) this is believed to be a valid conjecture. It is unknown which formatting engine has the second largest database publishing market share. There are database publishing systems that work with: ➤ FrameMaker ➤ Interleaf ➤ PageMaker ➤ Ventura ➤ Word Using FrameMaker for database publishing FrameMaker is one of the first desktop publishing applications that was used for developing database publishing systems. Early systems appeared in 1989 and used maker interchange format (MIF) as a means to tag information extracted from databases. Albeit crude by today’s standards, this methodology of database publishing presented a leap beyond main frame computer-based database publishing techniques. In 1993, Brio Publish created an off-the-shelf application for performing database publishing. This is the first known instance of a product created to work with FrameMaker expressly for database publishing. ➤ Has excellent output capability (PostScript, PDF, HTML, XML (soon), and word processors) ➤ Superior table capability Database published documents tend to be long documents. Whether you consider catalogs or directories, they tend to be hundreds if not thousands of pages long. If you use an engine that doesn’t have long document capability, you become trapped with non-continuous flow issues and discreet file management issues. To create mission critical applications or products for database publishing, a publishing engine must have an API that will enable the interprocess application (the application between the data sources and formatting engine) to control the application, obtain feedback, and populate the application with database information. Fortunately Framemaker has one of the best publishing engine APIs available. It can be used in conjunction with a variety of programming environments and it provides access to core publishing engine controls. This may be FrameMaker’s greatest unknown asset, its FDK. Why is FrameMaker the best? FrameMaker is the best engine to use for database publishing for several primary reasons. ➤ One of the best available engines for long document production ➤ Has an excellent application programming interface (API) called the Frame Developer’s Kit (FDK) 7 1998 FrameUsers Conference: Using FrameMaker in the Real World FrameMaker has excellent output capability. This allows a database publishing application to create products that can be distributed in paper or a variety of different types of electronic forms. Using one engine to produce many forms of the same information is very efficient. Tables are crucial to numerous types of database publishing projects. For example, catalogs typically use table to segregate product information. FrameMaker’s superior table capability enhances its ability to perform database publishing. In addition to these key reasons, there are a number of secondary reasons for using FrameMaker for database publishing projects: ➤ Book functions ➤ Multiple master pages ➤ Information cross-referencing and hyperlinking ➤ Automatic table of contents and indices ➤ Hexadecimal codes for special character insertion The other is to apply an off-the-shelf solution to database publishing problems that fall within an applications capability. Both approaches have been used by solution providers to implement successful database publishing systems. As development companies gain more mission critical solution experience, they will develop more generalized approaches to the database publishing problem. These solutions’ capabilities will grow and satisfy a larger share of the market needs. Methods There are two general process methodologies that are used for database publishing: ➤ Database dump to flat file that is manipulated and imported into Framemaker ➤ Application that provides a direct connection to the database engine and FrameMaker The process for the flat file importation process is show below. The process is implemented by developing a standalone application to dump information into a flat file that can be tagged for use in a pre-FrameMaker application processing application or into MIF. Formatting is typically controlled by using a tagged language. Mission critical versus application When providing database publishing solutions, there are two approaches that are generally used. One is to develop a solution that is specific to the problem being addressed. Stand-alone Application Database Flat File MIF or other meta-language 8 Database Publishing with FrameMaker: Future The process flow for the direct connection is show in the figure below. The direct connection uses a GUI to allow the user to setup and publish from the database. Formatting and operation is controlled by the user through the GUI and uses the FrameMaker template catalogs. ODBC Direct Connect SQL Objects GUI FDK ➤ PublishRight ➤ Refined Reports There are other companies that use FrameMaker and custom programming to provide database publishing solutions including: ➤ Business Brothers Software Products ➤ Finite Matters Ltd. ➤ Guidance ➤ SAIC The database publishing application industry is an immature sector. This is based on the following perceptions: ➤ No major players (e.g., Adobe, Microsoft, IBM, etc.) ➤ Base application price is high ➤ No industry tracking ➤ No major products with large share FML believes that this won’t change until a new idea is introduced into the market which rapidly captures market share and generates its shareholders profits. Database The players There are a number of companies that use FrameMaker as the formatting and output engine for database publishing. The following companies advertise products that work with FrameMaker for database publishing: ➤ Database Publishing Solutions ➤ Datalogics ➤ Datazone ➤ Finite Matters LLC Future FML believes that the demand for database publishing will grow as new applications minimize setup and learning and are low cost. Looking at the database management system market for NT and UNIX applications in 1995, we find that the total market value was $1.85 billion.1 During the same year, there were 4.4 million desktop database units shipped.2 This indicates that there are a lot of databases installed in the marketplace. If only 1% of the 1995 installed database market wanted to perform database publishing, it would be a large number of potential customers. 1. Informationweek, April 8, 1996, page 14, from Gartner Group. 2. Informationweek, February 19, 1996, page 15, from Computer Intelligence Infocorp. 9 1998 FrameUsers Conference: Using FrameMaker in the Real World Dr. Martha E. Williams, a professor at the University of Illinois Computer Science Department, has studied the database publishing market since its inception, writes: Often search intermediaries complain about the cost of online searching and tend to think that all database producers are making money at their expense. Few understand the high expenses associated with database production: that most databases do not enjoy a high enough level of use to produce sufficient online revenue to support their creation, and that a database is often largely supported by the corresponding print product. The foregoing statistics on database use and revenues should make these facts of life quite clear. Desirable as it may be to have an abundance of free information, as is available currently through the Internet, there still remains the need for quality information with the type of added value that many of the abstracting and indexing publishers as well as the publishers of numeric and transactional databases provide. The collection of information and the adding of value is done at considerable cost to database publishers and they must be compensated.1 nies will produce applications, which will most likely be purchased by the large software application manufacturers for distribution in the commodity software market. Some companies may develop applications that create a market sector of their own. That is the applications may work with multiple databases and multiple desktop publishing applications and in addition import information from HTML and other flat files. New features We expect new database applications with features and functions to be introduced to support database publishing of complex products. Integration with other applications To enable additional functionality, the database publishing will be developed to permit integration with other applications. For example, integration with scripting language would provide users the capability to write custom processing scripts to work in conjunction with the application. Other integrated applications may perform: ➤ Database clean-up In recent follow-up discussions with Dr. Williams, her conclusion hasn’t changed. This theory is applicable to publishers and others who add value to the information they collect. Projects Looking at current database publishing market trends, we can begin to clear some of the fog from the future. More companies are looking for tools to complete database publishing projects. The explosion of database engine installations has produced a need to publish that information for reading in a logical and user-friendly manner. More companies are developing database publishing solutions, with most of them centered around the WWW. As more companies develop database publishing applications, they will become more common as the product pricing falls. This is the general practice of increasing returns that is well practiced by Microsoft. Several compa- ➤ Database calculations ➤ Graphical representation of data These applications will allow the database information to be presented in its optimum presentation form. Two-way communication Customers are requesting the ability to not only extract information from databases, but also to import information from FrameMaker to the database. This is based on the concept of using FrameMaker as a front end to a database. Implementation of this would allow extracted, formatted FrameMaker data representations to be changed and the associated database information updated. Current thinking suggests that this may be possible on a case-by-case basis and that development of a generic application may not be possible. 1. Dr. Martha E. Williams, Publishing Research Quarterly, Fall 1995 10 Database Publishing with FrameMaker: Additional References In 1993, prototype systems were developed that used FrameMaker in conjunction with a database to accomplish similar functionality. A combined front end interface was developed that updated both the database and FrameMaker simultaneously. A similar application could be developed if FrameMaker and the FDK evolve. Integration with document management systems It may be desirable to integrate the database publishing with document management systems like Documentum, FileNet’s Saros Document Management System, and PC Docs. This would allow documents to be automatically checked into repositories after they are created. This will allow document management systems to control database published documents created from the results of different queries. For example, some information distribution systems may need a series of electronic documents created from databases. The document series may be based on different database inquiries, each document having its own query. A document management system could be used to store the series of documents to allow control and easy retrieval. Speciality Database Publishing Applications Market driven needs may suggest the creation of different database publishing applications for each sub-market segment. This is opposed to the current goal of developing one database publishing application that will fit the needs of all of the publishing market. Database publishing specific market applications could be developed for: ➤ General directories ➤ Phone books, including yellow pages ➤ Catalogs ➤ Financial reports Another concept that might have a market basis is to offer a combined FrameMaker/database publishing application, and Microsoft Access kit with pre-built database publishing systems. An architecture could be produced for specific business or organization functions and include pre-constructed items such as database tables, variables, and FrameMaker templates. This would allow users the capability to easily input or import information into a prebuilt sy stem an d use it witho ut h avin g to cr eate FrameMaker templates, formulate database structures, or database publishing application setup. Users could rapidly start using database publishing for specific cases. For example, a pre-built package could be created for associations to publish member directories. It could include pred e fin ed d a t a b a s e t ab le s , a p p l i c at i o n s e t u p , a n d FrameMaker template just to publish two or three columns per page directories that would include member name, address, phone number, and other desired information. Another example may be a package for simple catalogs. Multiple database publishing setups should also be included in a more generic product. This would serve two purposes; 1) to allow generic publishing process customers to get into production quickly and 2) to provide examples to customers that have to create more involved systems. The WWW WWW-based database publishing may present one of the largest market growth opportunities. As previously described, this would entail using a WWW interface to drive a database publishing session. This is a very powerful marketing tool which may find implementation opportunities. Additional References 1 Database Publishing with FrameMaker, Frame Technology Corporation, 1992 2 Database Publishing White Paper, Finite Matters Ltd., 1998 Trademarks FrameViewer, FrameReader, FrameViewer Pro, FrameMaker, FrameMaker+SGML, Frame Developer’s Kit (FDK), PostScript, Illustrator, Adobe Type Manager, and Acrobat are registered trademarks of Adobe Systems Incorporated. 11 1998 FrameUsers Conference: Using FrameMaker in the Real World Apple and Macintosh are registered trademarks of Apple Computer Corporation. PatternStream is a trademark of Finite Matters LLC. ARMS, FrameTools, and FrameScript are trademarks of Finite Matters Ltd. Excel, Visual Basic, Visual C++, Windows 3.1, Windows 3.11 for Workgroups, Windows 95, Windows 98, Windows NT, TrueType, Access, SQL Server, BackOffice, Site Server, and Jet Database are registered trademarks of Microsoft Corporation. All other brand or product names are trademarks or registered trademarks of their respective companies or organizations. 12