HOWTO: Examining Datafiles with Metakit

Last modified 2/10/2006.

CourseForum and ProjectForum rely on the open source database storage library Metakit to manage most of their storage needs. All the ".db" files you'll find in the "Course Data" or "Group Data" directory are Metakit datafiles.

Not only does the software use Metakit internally, but you can use it as well to examine (or perhaps, very carefully) modify the datafiles. There are simple command line and graphical tools availiable for browsing the databases, as well as programming API's in C++, Tcl and Python.

Using SDX to Examine Datafiles

A useful command line tool is 'SDX', which runs on a variety of different computing platforms. This lets you easily look inside Metakit datafiles.

To use it, you need to download two things. First, you need a program called tclkit. There are versions for many different platforms. Second, you need to download sdx.kit, which is a platform-independent utility that can be run by tclkit. More info is available on the sdx info page.

Once you have those two files, you can do things like:

tclkit sdx.kit mkinfo course1.db
(which tells you the structure of that database file)

tclkit sdx.kit mkshow course1.db attachments
(which dumps out one table in that database file)

The "mkinfo" command may produce something like the following; the first number on each line indicates the number of rows in each table, identified by the second word on each line. The rest of each line details the fields stored for each row in the table.

course1.db
  Metakit data starts at offset 0 and is stored as little-endian
  Views:
   1134x current {pageid:I forumid:I created:I modified:I versionnum:I name 
         username body lockpassword postwhenlocked:I posting feedid}
      1x currentctl nextid:I
    657x recent {forumid:I pageid:I name modified:I username}
   1665x links {from:I to:I}
     75x options {id:I name value}
  10085x versions {forumid:I pageid:I modified:I versionnum:I username body}
     22x accounts {username password passwordcookie}
      1x accountsctl nextid:I
  60717x activity {forumid:I pageid:I versionnum:I timestamp:I operation username}
   1229x attachments {attachid:I forumid:I pageid:I filename originalfilename 
         contenttype timestamp:I category}
      1x attachmentsctl nextid:I
      1x notify {email forumid:I pageid:I frequency:I lastnotified:I}
      0x pendingnotifications {forumid:I pageid:I notifytime:I email}
      1x globalctl formatversion:I

The "mkshow" command might display something like this (in this case using a different database file that only contains two rows in the attachments table):

   1435  1  9  attachments1/file1435  hostingterms.html  text/html  1084044043
   2608  1  9  attachments1/file2608  bsdlogo.gif  image/gif 1084044057  image