  RMS files, journaling, and BACKUP support? 
 The Question is:
 
How can we access and process the information stored in an OpenVMS RMS
Journaling file using a 3GL program? We basically need to know the FILE
FORMAT (spec) of OpenVMS RMS Journaling JOURNAL FILES.  (OpenVMS 6.2+)
 
We are attempting to create a data warehousing/management/transfer
system using existing OpenVMS RMS keyed database files by nightly batch
"mirroring" the data to another system/file/database format. (unix/relational)
 
To make the process work as productively as possible, in regard to speed
and accuracy, we are in need of transaction information regarding file
modifications to the RMS files.
 
Specifically, we need to know when a record has been updated, added and
deleted. We are only in need of the "raw" modification method, as opposed
to actual field or record changes. That is, that "a record" was added, deleted,
or modified, which specific record (RID?), and, if possible, when the record
transaction occurred -- date/time.
 
We have researched this problem and have not found any Compaq (DEC)
or third-party products that would help us solve our problem. Therefore, we
have come up with a solution that we believe would be the best way to solve
the problem, but we are in need of additional information.
 
Our solution is to use RMS Journaling "journal file" information to detect
when a record has been added, deleted, or modified. However, we need to
know what the journaling file format is, so that we can write a (VAX BASIC/C)
program to read and process the information, which will then be used to
provide incremental data transfers (reducing size/time problems in a multiple
large file system).
 
Can you provide us with the JOURNALING FILE FORMAT SPECification and/or
any other information that would help us in this regard? We could even use a
report output generated from the journaling file, parsing the text output to
extract the information we require, if something like that was available.
 
Any hints, tips, suggestions and other information provided by any persons
would be greatly appreciated, and may be sent via e-mail to Bernard Buckley
at: buckleybe@butte.cc.ca.us
 
Thank you.
 
 
 The Answer is:
 
  You do not and should not directly access the contents of the RMS
  journaling product's journal files.  The contents, operation, and
  format of RMS journal files are undocumented and subject to change
  without notice.
 
  If you wish to perform a rapid backup of your application, please look
  to coordinate the your application file system activities -- via lock,
  via the distributed transaction manager, or otherwise -- with a utility
  or tool based on the BACKUP API or other archival tool or product that
  is specific to the application.  This approach typically involves
  synchronous flushes of RMS caches via calls to sys$flush and synchronous
  flushes of global memory via $updsec[w], and/or (at its simplest and
  slowest) calls to temporarily close files via sys$close or similar,
  perform the BACKUP, and then re-open the files.
 
  Various products -- including Oracle Rdb -- utilize an approach that is
  based on an application-integrated BACKUP.
 
  OpenVMS uses an approach based on a lock for operations such as extending
  files -- when the volume lock is set via ANALYZE/DISK/REPAIR or other
  tool, various file activity incompatible with the analysis and repair
  is temporarily halted, while various compatible activity can continue.
  Once the lock is released, activity continues.
 
  One option for quick(er) BACKUP operations involves quiescing the
  application and flushing the buffers, and then splitting a volume
  off a shadowset -- minicopy and related work in OpenVMS specifically
  targets shortening the window of quiescence required for the BACKUP.
 
  Also consider acquiring a commercial relational database product.
 
  Also consider acquiring one of the ODBC/JDBC packages or another other
  distributed or served network-based approach -- please see the OpenVMS
  FAQ and various discussions here in the OpenVMS Ask The Wizard area of
  ODBC/JDBC -- and directly access the contents of the existing RMS file
  from the remote system(s).  This remote access can be performed via a
  commercial package, or via a an application-specific network server
  process.
 
  Note that the RMS Journal files currently do not maintain information on
  record modification operations, these files only contain the after images
  of the changed buckets as a result of the record modification.
 
  For example: Bucket number 6 in an index file is 6 blocks big, and has
  record with primary key A,B and C, for a total of 2300 bytes. Record B
  is now updated. The AI file will get a copy of the first 2300 (out of
  3072) bytes of bLOck 6 data containing old records A nd C and the new
  data for record B. There will be no indication what-so-ever that record
  B caused the change.
 
  To establish the record context, you would have to intercept the AI bucket
  as is was about to be appllied to a datafile backup and compare old and
  new data. This is clearly non-trivial.  Once you have achieved this, you
  you would acquire the date and time stamps as reward in the journal file
  record header data.
 
  The most expedient mechanism the OpenVMS Wizard can provide that would
  reconstitute the AI data back to record operations would require three
  copies of the file:
 
           1 - Live data
 
    	   2 - Master backup
 
    	   3 - copy of backup with AI file applied.
 
  Now write a program to compare file 2 and file 3
 
    Records found in 3 but not in 2 become a $PUT.
 
    Records found in 2 but not in 3 become a $DELETE.
 
    Records which are different become an $UPDATE.
 
  To handle this properly, you would probably need to monitor KEY and RFA
  values.  You would likely loose temporal insight, although you could
  choose to apply the AI file up until a certain timestamp.
 
  The OpenVMS Wizard would strongly recommend encapsulating RMS record
  operation calls within application routines (MY_PUT versus SYS$PUT), and
  those jacket routines would call onwards to RMS for the real operation
  and would log the changes in a sequential file.  This is still not trivial,
  but is feasible, supportable, and documented.
 
