Originally published in Handheld Systems, Vol 5.2

"No man is an island", they say, and neither is, in most cases, a PDA.  Especially when custom software is involved, there are numerous reasons why data has to be exchanged between a portable device an the rest of the world: making a backup copy of an organizer's contents, incorporating data gathered "on the road" into a larger bussiness process or downloading information for mobile use which was originally entered on a desktop PC. This article tries to give you some ideas on what you can do in your own applications to get data into or out of a Geos-based device.

Introduction

Although devices like the OmniGo 100 or the Nokia 9000 try to hide this fact from the user quite nicely, long term data storage on Geos PDAs is still based on a DOS-like file system, with each application creating individual data files within a directory structure that is fairly similar on all Geos-platforms.

As a consequence, data exchange with a Geos PDA usually consists of two relatively separate problems:

1. Moving individual files between the PDA and the "host" platform.
2. Reading and writing the content of these files. This requires special attention when working with a Geos device, because the system is offering a number of higher-level file structures, like VM files, which result in Geos files becoming fairly complex when looked at from a non-Geos application.

Of course, you could also think of a situation where the PDA and the "host" system directly negotiate data updates through a (serial) connection, but this will usually result in quite a lot of additional effort on both sides.

The transfer of physical files is somewhat specific to the device your application is running on (because it depends on the built-in communications capabilities), and it will be dealt with here especially with the OmniGo in mind. On the other hand, most of the peculiarities of Geos files will stay the same from one Geos platform to another.

This article generally takes the perspective of someone who has control over both the application on the OmniGo and the application on the "host" system.  If you want to make use of the existing OmniGo applications as a front end to your data, you will need additional knowledge about the HP Database Library (which is very similar to the one used in the 100/200LX series palmtops) and the way the built-in applications are using it - this may be the subject of a future article.

File exchange with the OmniGo

The OmniGo offers two different means by which it can exchange information with the outside world:

• the PCMCIA card slot
• the built-in serial port

While a serial port is probably available on nearly every machine you will ever want to connect your OmniGo to, data exchange with PCMCIA memory cards requires a special card drive in at least one host PC. On the other hand, copying large amounts data to and from memory cards is considerably faster than transferring them through a 19200 bps serial connection, and using the serial port quickly drains an OmniGo's internal batteries (especially when using rechargables).

PCMCIA memory cards

While data on a PCMCIA card is also stored in the format of a DOS-style file system, there is a special twist in the way Geos integrates these files with the built-in storage of the OmniGo: Even though the resident RAM disk and the card memory are assigned different drive letter (B and C, respectively) by the underlying DOS, Geos treats them logically as a single volume in some cases.

In particular, this applies to all files in the directory \GEOWORKS and below on both drives: if such a path exists on the memory card, any files and directories in it will seamlessly integrate with the corresponding paths of the built-in RAM disk: For example, if an application is stored in the \GEOWORKS\WORLD directory of the memory card, it will appear on the OmniGo's home screen as if it was stored in the WORLD directory of the internal drive.  This mapping is implemented through the file system of Geos, so any application may take advantage of it without addition efforts on the side of the programmer.

This offers a very elegant way of getting data from a PC into an OmniGo - on the PC's card drive, simply create a file structure that mirrors the standard \GEOWORKS tree (you'll only have to create those directories that you want to put data into, not the empty ones), and copy your data files to the proper location (often, this will be the DOCUMENT or USERDATA directory). Once the card is inserted into the OmniGo, any application can transparently access these files without caring about their location (internal RAM disk vs. memory card).

It also works the other way around (that is, from the OmniGo to the PC), as long as your application is only modifying existing files, in which case they will stay on the card anyway. If new files are created, you will have to specify the drive letter of the card (that is, C) explicitly to avoid them being created on the local RAM disk. You can then put the card into the PC's drive at any time (don't forget to shut down access to the card before using the F4 softkey in the Home screen) and process the modified/new files.

Using the Transfer application

If you do not want to have the memory card with the data inserted into the OmniGo all the time (or if you want to reduce the number of cards required when using multiple number of devices), you can still use cards for file exchange by taking advantage of the backup function built into the OmniGo's Transfer application.

When a card is inserted, the right column of the Transfer screen will show a list of data backups made to that card, if any. After selecting a file on either side, you can use one of the "Copy" softkeys to backup data to the card or restore data from a previous copy.

Backup copies are stored in a special directory tree on the memory card: immediately below the card's root, there is a directory named @TRNSFER.000.  This main directory in turn contains subdirectories for every backup made to the card, with one "backup" corresponding to one entry in the Transfer screen.  These directories are normaly called something like 76D00000.BCK (i.e.  automatically generated "temporary" names), but their name doesn't really matter, because Transfer will recurse through the structure anyway.

The backup directories may each contain any number of files (for example, all databases on the internal RAM disk can only be backed up at once), together with one special file called @APPNAME.000, which is a plain DOS file containing the name of the application from which the data was backed up (zero-padded up to a length of 36 bytes). The name can in fact be any descriptive text which will be displayed on the transfer screen.

With this knowledge, any PC-based application can also use a memory card drive to create data that is accepted by the Transfer application as a "backup", which means that it can be easily copied to the internal RAM disk using Transfer. If a large enough memory card is used, an application can also create multiple "backups" which can then be copied selectively to the OmniGo.

To transfer an application's data files from the OmniGo into the backup structure of a memory card, this application must first make itself (and its data files) known to the Transfer application, so its name will appear in the left column of the main screen if data files are present (note that these files must be located in the DOCUMENT directory to be offered for transfer).

Geos files (as opposed to plain "native" DOS files), which can have a long filename and may carry extended attributes are automatically recognized based on their "creator token" attribute (specifying the token of the application which can deal with the file). This attribute can be set using the FileSetPathExtAttributes() API call after the file was created. For example, any file compressed with the COMPTOOL sample application from PDA Developer's magazine 4.5 automatically causes a "Pack Files" entry (the screen name of the creator application) to show up in the Transfer screen. Again, note that backing up this entry will cause all compressed files to be copied to the card.

Native DOS files do not have a creator token, so another way must be used to assign them to a specific application: Tokens for them can be defined based on their file names using the filenameTokens entry in the [fileManager] group of the GEOS.INI configuration file. When looked at with a text editor, this entry, which is a so-called "String Section Blob", looks similar to the following...

[fileManager]
filenameTokens = {
  *.HTM = "gDAT",0,"HMLV",16431
  *.GIF = "gDAT",0,"HMLV",16431
  *.NDB = "gDAT",0,"NOTE",0
  :
}

This entry assigns to any file matching the file specification on the left side of the "=" sign the two tokens on the right side. The first token can always be "gDAT",0 on the OmniGo (on other platforms, it will specify the Geos file type and hence the icon which represents the file). The second is the token of the application which is regarded as "creator" of this file by Transfer. The above sample assigns any HTML and GIF files to the application with the token "HMLV",16431, while NDB files are listed under the name the Notepad (in fact, this last entry is already part of the OmniGo's default configuration).

An application wishing to add itself to this list can use the InitFileWriteStringSection() API call to append a new entry. This should either happen only once when the program is installed on the device, or it should use InitFileEnumStringSection() to check if the entry already exists to avoid creating duplicate entries.

Using the Install Manager

If you are only looking for an easy way of downloading application-generated files to the OmniGo, creating package files for the Install Manager may be an option. This has the advantage of not having to implement the file transfer protocol yourself, and besides it offers some flexibility by permitting the user to choose between downloading package content to the internal RAM disk or a memory card in the OmniGo as well as using a card drive in the PC the Install Manager is running on. In addition, the Install Manager can also be used to delete packages from the OmniGo again.

The obvious disadvantage of this method is that it is a one-way street, in other words, it cannot be used to get data entered on the OmniGo back into the PC.

The GPK package files used to pass content to download to the Install Manager are actually quite simple in their structure, so creating them from your own application (instead of having to go through the "Shipper" tool every time the content to be downloaded has changed) will not be very difficult:

Every GPK file starts with a header that can be described by the following structure:

struct GPK_header {
  char           ID[8];       // = "GEOS_GPX" (format id)
  long           _x1;         // = 1 (probably format version)
  char           device[4];   // = "JEDI" (probably target device)
  long           _x2;         // = 1 (device version?)
  unsigned short n_files;     // number of files in the package
  long           pos_title;   // file offset of title
  long           pos_desc;    // file offset of description
  long           pos_file[];  // repeated n_files times
};

pos_title contains the file position of a (zero-terminated) string specifying the name of the package displayed in the Install Manager. pos_desc points to a string that is shown as a package description when the user clicks on the package name. Finally, the pos_file[] array contains the file position of the header records for each of the package's files:

The header record starts with three consecutive variable-length, zero-terminated strings specifying the filename (for example, TEST.000), the fully qualified destination name (for example, \GEOWORKS\DOCUMENT\TEST.000) and the version number of the file. The third string is followed by a long value giving the size of the file in bytes.

If the size of the file is non-zero (zero length files can be used as "placeholders" for later removing files which are later by the application on the OmniGo), the size value is followed by another long value specifying the number of blocks containing the file data.

Each block starts with a long value specifying the number of data bytes in the block (I always found this to be 1FF8h, except for the last block of a file which may be smaller), followed by as many data bytes as indicated by that number. The next block, if any, will follow immediately after that.

The source code disk contains a simple command line program named GPKMAKE which uses a human-readable ASCII list of files to automatically generate a GPK file.

Two-way connectivity

If you want to transfer your data to and from the OmniGo over a serial connection, there are generally three options, all of which use the Transfer application as a "communications server":

• Use a general-purpose file transfer tool like OmniCom, OGFTP or the new OmniGo Remote Controller (ORC) to get data back from the OmniGo. This gives you great flexibility, but it will usually be hard to perform this task automatically if frequent updates are required.
• Call the PCGET and PCSEND utilities from your PC application to do the transfer work. These two programs are included with the SDK, and according to the terms contained in \LICENSE.TXT on the SDK CD, they may be freely distributed together with your application. For example, you could use the command "pcget /b:9600 document\notepad.ndb" to download a copy of the notepad database from the OmniGo into the current directory. For this to work, the OmniGo's Transfer application must be in "Connected to PC" state.
• If you want to fully control the transfer process, you can also add support for the OmniGo's PCCOM file transfer protocol directly to your application. While the protocol is not extremely complex and is also fully described in the "Tools" book of the SDK, this approach will still require you to get into communications programming on your chosen host platform.

Content exchange with Geos applications

One of the interesting features of the Geos operating system is that it is offering a number of storage methods which go beyond the simple random-access byte level files found on other platforms. As these concepts are used throughout the system and generally facilitate development, many Geos applications will store data using these system-specific enhancements.

While this makes the job easier for developers of Geos applications (once you have come across the steep end of the learning curve for any new feature), access to Geos files from a non-Geos platform requires additional effort and knowledge about the way in which things like VM (Virtual Memory) files and long filenames are mapped to ordinary DOS files at the byte level.

The following sections describe the common features of all Geos files when looked at from a non-Geos perspective, as well as the specific implementation of VM Files, which form the base of the more advanced features.

Of course, the data stored in the files is ultimately determined by the application creating it, so you will have to add more application-specifc knowledge on top of the common structures to be able to parse a Geos file. In addition, system data structures not covered here (Local Memory heaps or Chunk Arrays, for example) can often be found in the header files of the Geos SDK (like lmem.h or chunkarr.h).

As most of the following information is not covered explicitly in the SDK, I have written a DOS program named GEODUMP which can create structured dumps of most file formats used by Geos, including VM files, executables and font. The program has helped me many times in the past to track down bugs in applications or to clarify statements made in the SDK documentation... 

By the way: the Geos SDK itself is a good source of such "undocumented" information if you know where to look. For example, a good way for finding out about data structures is to use the PRINTOBJ utility (which itself is undocumented) on the SYM file of a library and redirect its output to a file.  This will present you, among other things, with a list of all the structures used in the library, together with their member names and types - if you can think of a keyword to look for, the names alone will often give you half the answer already.

Of course, this information should not be used to create "hack" type solutions while there is a better option offered by the system itself, but in somes cases it can add that last bit of understanding that is required to use a valuable feature.

The common file header

Every Geos file starts with a file header of 256 bytes which contains all those extended attributes that cannot be stored by DOS itself (that is, nearly all of them). If you want to create "pure" files which do not even contain this part, pass the FCF_NATIVE flag to the FileCreate() call.

The header can be described by the following structure:

struct GEOS_header {
  long           ID;               // = 0x53CF45C7 ("magic")
  char           name[36];         // long file name (ASCIIZ)
  unsigned short type;             // Geos file type (word sized)
  // SDK type: GeosFileType        //  1 = executable file
                                   //  2 = VM file
                                   //  3 = byte level data file
  unsigned short flags;            // attribute flags
  // SDK type: GeosFileHeaderFlags //  0x8000 = template file
                                   //  0x0800 = hidden file
  ReleaseNumber  release;          // release number
  ProtocolNumber protocol;         // protocol number
  GeodeToken     token;            // token of file type
  GeodeToken     creator;          // token of creator application
  char           info[100];        // user file info (ASCIIZ)
  char           copyright[32];    // copyright notice
  PackedFileDate create_date;      // creation date/time in DOS format
  PackedFileTime create_time;
  char           password[8];      // password, encrypted
                                   // all zeroes: no password
  char           _reserved[44];    // reserved, should be 0
};

The ReleaseNumber structure contains four words, specifying major, minor, change and engineering version number of the file, or the application that created it. These numbers are usually written down the form of "1.0 15-54", but they normally have no effect on the handling of a file, except for the checking performed by the Swat debugger to ensure that the version of a symbol file matches that of an associated exectuable.

The ProtocolNumber structure contains two words, specifying major and minor protocol number of a file. These values determine the compatibility of a file, for example a system library - libraries which differ only by a larger minor version number contain a compatible superset of functions, while differences in the major version number indicate incompatibility.

A GeodeToken consists of four characters, followed by a word-size manufacturer ID (such a number is assigned to developers by Geoworks at request and serves as a unique identifier for some user-defined types in the system). The creator token links a data file to an application that can handle it (see previous discussion of the Transfer application for usage of creator tokens on the OmniGo).

The copyright notice, which is normally only visible by direct inspection of the file's binary representation, can be set for an executable file at compile-time by adding the line

XLINKFLAGS = -N your\20Copyright

to the LOCAL.MK file in your project's root directory (if the file does not yet exist, make sure that you also include the line #include <$(SYSMAKEFILE)> in the file. Otherwise, a default copyright notice of Geoworks will be embedded.

The create_date and create_time fields are packed structures using the standard DOS representation of time stamps:

typedef struct {
  unsigned short d:5;
  unsigned short m:4;
  unsigned short y:7;    // years since 1980
} PackedFileDate;

typedef struct {
  unsigned short s_2:5;  // seconds/2
  unsigned short m:6;
  unsigned short h:5;
} PackedFileTime;

Byte level files

If the type field of the header contains the value 3, the file is a byte level data file which contains no additional structure provided by the system.  Such a file can be created using the FileCreate() call. Any bytes following the header are identical to what has been written by the application.

When parsing such a file on a non-Geos platform, the only thing that has to be observed is that you must add 256 to all absolute file positions that may have been retrieved using FilePos() under Geos, because from a Geos perspective, the header is not really part of the file.

VM Files

If the type field of the header contains the value 2, indicating a VM file, the following 24 bytes are interpreted as an additional header

struct GEOS_VM_header {
  unsigned short IDVM;     // = 0xADEB ("magic")
  unsigned short dirsize;  // size of directory block (bytes)
  long           dirptr;   // absolute file position of directory
  char           _x[16];   //   (reserved)
};

The basic structure of any VM file consists of a series of data blocks with a maximum size of 64k each. Every block in a file is identified by a word-sized block handle (corresponding to the SDK type VMBlockHandle). If a VM file is used under Geos, these blocks are temporarily attached to memory blocks on the global heap whenever they are loaded into memory for proessing.

While a VM file is open, the operating system may freely move around these blocks within the file to avoid wasting space when blocks are deleted or changed in size. Normally, this happens especially during "compacting" events when the whole file is rearranged to removed accumulated slack space.

The position contained in dirptr points to the start of a block which in turn lists all the blocks contained in the file, including itself. Again, this position is based on the byte immediately following the common Geos header, not the start of the file.

At the beginning of the block, the following 32-byte-structure conveys general information about the entire file:

struct GEOS_VM_dir_header {            
  unsigned short IDVMdir;        // = 0x00FB ("magic")
  VMBlockHandle  hdl_1stfree;    // handle of first free block
  VMBlockHandle  hdl_lastfree;   // handle of last free block
  VMBlockHandle  hdl_1stunused;  // first unused handle entry
  unsigned short dirsize;        // total size of directory block
  unsigned short nblocks_free;   // number of free blocks
  unsigned short nhdls_free;     // number of unused handle entries
  unsigned short nblocks_used;   // number of used blocks
  unsigned short nblocks_loaded; // number of loaded blocks
  char           _12[2];         //   (unknown)
  VMBlockHandle  hdl_map;        // handle of map block
  unsigned short compaction;     // compaction threshold, 0 = default
  long           totalsize;      // total size of allocated blocks
  unsigned char  flags;          // VMAttributes
  unsigned char  _x2;            //   (unknown)
  VMBlockHandle  hdl_dbmap;      // handle of DB map block
};                    

Some of this information, like the number of loaded blocks, is only valid while the file is opened. Most others have mainly administrative value and do not have to be observed when reading a file created by Geos.

Normally, the most important field in the header is the hdl_map field. Its content can be set using the VMSetMapBlock() API call, and it is intended to record the handle of an application-defined header block which can serve as a "starting point" when retrieving data from the file. (Otherwise, as all information in a VM file is completely dynamic, there would be no fixed location from which data could be initially loaded.)

The header is followed by a list of 12-byte long block directory entries. Any VMBlockHandle is in fact a word-sized offset from the beginning of the directory block to one of these records (that is, the lowest possible handle value of 0x20 points to the location immediately after the directory header).

The block list contains two different types of entries: those representing blocks filled with data and those tracking free space. Handles for allocated blocks point to a structure of the following form:

struct GEOS_used_entry {
  MemHandle hdl;          // handle of block if in memory
  unsigned short flags;   // flags for this block
                          //  0x0100 = block contains LMem heap
  unsigned short userID;  // user-defined identifier
  unsigned short size;    // size of block
  long blockpos;          // file position of block
};

If blockpos is 0, the handle is unused. The userID field contains a value that can be defined by the application (except for values of 0xFF00 and above, which are reserved for system use, and the value 0xADEB, which is used for the directory block).

An entry in the chain of free blocks looks like this:

struct GEOS_free_entry {
  VMBlockHandle  next;    // handle of next free block
  VMBlockHandle  prev;    // handle of previous free block
  long size;              // size of free area
  long blockpos;          // file position of free area
};

With this information, data stored in VM files can be accessed from non-Geos applications. Anyway, as you can see, dealing with VM files is far from trivial, so it may often be more adequate to simply create a special transfer file format based on native DOS files for getting data into and out of your OmniGo application.