WebMO Help - WebMO Database Files

Database Files

WebMO maintains three internal databases in the {userBase} directory with information about jobs, users, and groups:

These files contain the authoritative information about the status of WebMO jobs and users.

Index Files

In order to speed up the display of information in Job Manager, particularly on systems with large numbers of jobs, the database files are pre-indexed by user. These index files are saved in the {userBase} directory as:

Viewing Database Files

Within the WebMO distribution, there is a directory called WebMO.install/scripts which contains utilities to display the database files. To view the contents of the databases, open a terminal session and run
$ cat_jobs.pl {userBase}/jobs.db
$ cat_users.pl {userBase}/users.db
$ cat_groups.pl {userBase}/groups.db

Regenerating Index Files

If an index files does not exist when it is needed, it is automatically regenerated. Thus to regenerate an index file, simply delete it and invoke the Job Manager. The .*.index will be regenerated automatically from the corresponding *.db file.

Editing Database Files

Important: Editing database files is not a normal operation supported by WebMO. Any editing should be done with extreme caution, as the database files contain the authoritative information about WebMO jobs and users. The *.db files should be backed up before any work is done on them.

Under normal operation, the database files should not be come corrupted, as information is only written to the files, not deleted from them. However, circumstances can arise, e.g., if the hard disk were to fill up or if file-locking is not implemented by the operating system, that could lead to their corruption. (This is why the /scratch directory is usually placed in its own partition.)

If a database needs to be edited, the following procedure can be attempted:

  1. Backup the *.db file
  2. Use the db2xml.pl script to convert the *.db file to XML *.xml file
  3. Edit or remove the offending lines (typically at the end)
  4. Use the xml2db.pl script to convert the *.xml file back to a binary *.db file
  5. Delete the corresponding .*.index file, which forces it to be regenrated from the newly edited *.db database file

A completely unsupported but very effective method for fixing a corrupted jobs.db file is to edit it directly. It is a binary file with fixed record length files, but each record is terminated by a newline. Thus one can open up the file in vi, find the last good record, and delete the subsequent few records to get back to a "good" place.

Database corruption is almost ALWAYS caused by the disk being full and failing to fully write the last line in the database file. Deleting the last incomplete line usually fixes the problem!

Purging the Jobs Database of Deleted Job Records

When jobs are permanently deleted by the user, the job record remains the the jobs.db file but is marked as deleted. If large numbers of jobs have been run and then deleted, this can lead to a very large jobs.db file. Deleted job entries can be removed from the jobs.db file by running
$ clean_jobsdb.pl {userBase}/jobs.db
Because the jobs.db file is so important to WebMO, a jobs.db.bak is automatically created of the jobs.db file before the record deletion operation is attempted.