Daylight THOR-Merlin Manual

Copyright Notice

This document and the programs described herein are Copyright © 1992-2011, Daylight Chemical Information Systems, Inc., Laguna Niguel, CA. Daylight explicitly grants permission to reproduce this document under the condition that it is reproduced in its entirety including this notice, and without alteration. All other rights are reserved.

Table of Contents

1.1 Configurations
1.2 The Server
1.3 The Client
1.4 Server and Client Applications

2.1 Starting Servers
2.2 Stopping Servers
2.3 Auto-loading Databases
2.4 Multiple Servers
2.5 Sending Messages to User
2.6 Monitoring and Maintenance

3.1 Starting Servers
3.2 Stopping Servers
3.3 Auto-loading Databases
3.4 Multi-CPU parallel processing
3.5 Multiple Servers
3.6 Sending Messages to User
3.7 Monitoring and Maintenance

4.1 Security and the Operating System
4.2 Server Security
4.3 Database Security

5.1 Database Creation and Maintenance

5.1.1 Creating and Loading a New Database
5.1.2 Loading New Data
5.1.3 Dumping a Database
5.1.4 File Ownership and Privileges
5.1.5 Moving Databases
5.1.6 Using Multiple Disks
5.1.7 Backing Up and Restoring Data

5.2 Record Locking
5.3 Read-Only Databases

6.1 Memory Usage
6.2 Loading a Merlin Pool
6.3 Columns and Cells
6.4 Column-creation functions
6.5 Loading Pools from Alternate Sources

7.1 Command-line Options
7.2 Daylight Options
7.3 Navigating
7.4 Functionality
7.5 Batch Processing

8.1 THOR Programs
8.2 Merlin programs
8.3 Generic programs
8.4 Entering Database and Server Names
8.5 Generic Options

1. Networking

THOR and Merlin are powerful tools for maintaining and exploring chemical databases. In this chapter, we explore the client/server network architecture that THOR and Merlin use; the chapters that follow describe the chemical-information aspects of THOR and Merlin. Both THOR and Merlin are client/server systems and the following will be applicable to both; where there are differences it will be noted. In general, a typical Daylight database system will involve the two servers, thorserver and merlinserver, and one or more client program which connects to one or both types of servers (e.g., XVMerlin is both a thorserver and merlinserver client).

As the name implies, a client/server system actually runs as two programs: a client (an application program), and a server (a program supplied by Daylight). The words "client" and "server" give you some idea about the relationship between the two: The server provides a service (database access), and the client uses that service.

The server's role is relatively straightforward: It has exclusive access to all databases that it opens, and performs all database transactions. Clients never access a database directly, but rather access all data via a server. The server can serve many clients at once, so normally only one server runs at a time on each computer, no matter how many clients are running.

The client's job is somewhat more complex. Clients have to deal with users (who are notoriously unpredictable), communicate with the server, generate unique SMILES for use with the server, and other such tasks. Each client can connect to many servers, and each client can open many databases on each of the servers.

Normally the client/server design is nearly transparent to the application. Given a server's "name and address" (using standard network protocol), the Daylight system takes care of connecting to the server via an Interprocess Communication (IPC) mechanism, and handles all communication with the server. When a client requests a database transaction, the Daylight system translates it into a request to the server; the message is sent via IPC, the server responds via IPC, and the Daylight system translates the response back into the answer to the client's request.

Because clients and servers communicate via network protocol, the client and server can reside on different machines or on the same machine. The IPC mechanism is designed to work equally well either way; once a connection to a server is established (a procedure that is the same no matter where the server resides), the client doesn't have to be concerned about the server's actual location.

The client/server mechanism uses a standard networking mechanism to communicate, one that is independent of the operating system of the machines on which the client and server are located. This means that heterogeneous mixtures of machines can be used to run a Daylight database system. For example, a large Sun system might run a server program; this server could serve a number of SGI workstations situated on a local-area network (LAN).

Some other advantages of the client/server design are:

• Guaranteed uninterruptible operations The server completes each transaction before it responds to a request from another client. There is no need for file locking, record locking, or other exclusive-access mechanisms; only one program ever accesses a database.

• Networkable The server can be placed on a big, fast machine with googlebytes of disk and memory, serving smaller workstations around a laboratory (or at remote sites if a wide-area network is available). The cost of expensive equipment (the disk, memory, and high-speed computer), as well as valuable data, can be shared among many clients.

• Caching With a single server, it is possible to cache (store in memory) parts that are accessed frequently. Caching is nearly impossible in "traditional" systems where multiple programs have to simultaneously access a database, due to synchronization problems.

• Access control The server can, at any time, list the users of a particular database. It can also lock a database for exclusive access even when other users already have it open, and other such functions that require complete control over who is using a database.

• Database integrity The server is a relatively simple, robust program. When a client program has a problem, such as crashing due to a programming errors during debugging, the server detects the problem and simply closes the database. Similarly, because all requests go through the server, which verifies that each transaction is legitimate before executing it, it is virtually impossible for a client to corrupt a database.

• Security Only the server process has direct access to databases. Security is greatly simplified and improved since all client programs must pass the server's scrutiny.

Of course, the client/server mechanisms do carry a modest price:

• IPC overhead For each transaction, the client must communicate the transaction to the server, adding to the overhead of each database access. This overhead is usually unimportant, since the time it takes for the client and server to communicate is small compared to the time it takes to read data from and write data to the database. If a central, super fast, networked server is available, it may actually be faster to ask it questions than it would be to access a database directly.

• Administrative complexity Some network administration is required, even in installations where the client and server will run on the same machine.

1.1. Configurations

In the figure "Distributed Databases" (below), every machine contains one or more databases, and runs a server and one or more clients. Each client has access to all databases on all machines, thus forming a system in which the data and computing burden are spread across the whole network. This might be useful in a research laboratory in which a number of researchers independently collect data. Each researcher would be responsible for the data in one database, yet all researchers would have access to all data.

In the figure "Centralized Databases" (below), a central machine contains all databases and runs the server; all data are accessed by remote clients that connect to the central server. A scheme like this might be useful in a large corporation with a central database in which all corporate chemical information is stored. The central server would most likely be a large, expensive computer capable of handling large numbers of requests.

1.2. The Server

As mentioned earlier, a server is a relatively simple program. In computer lingo, it is a daemon- a process that runs in the background with no user interface of its own. Since it has no user interface (no terminal or window into which you can type commands), all communication to a server is done via client programs.

Each server keeps a log file of its activities. This file contains a record of certain selected client activities, such as client connect/disconnect events, security information, database openings and errors. The actual file's name is configurable; see the chapter on configuring and running servers for more details.

The following is a partial list of a server's responsibilities:

• Accept connections from clients; create a client context in which to store the names of databases the client has open and other client-specific data.

• Detect lost connections; close all databases that the client has open and discard the client context.

• "Ping" - echo a message back to a client (test a connection).
• Share database resources among clients (caching, open files, disk space, cpu). For example, a database that is opened by two clients doesn't use any more resources than one opened by a single client.

• Store and retrieve data.

• Rebuild databases that are damaged, "garbage collection".

• Provide information about databases (sizes, who is using it, etc.)

• Security. This is discussed in more detail below.

1.3 The Client

As we noted previously, a client is a more complex beast than a server. A server only has to deal with a well-specified set of requests from client programs. By contrast, client programs typically deal with users, who are unpredictable and demanding; client programs are correspondingly more complex.

Unlike a server, which is a single program supplied by Daylight, a THOR client is any program that uses the THOR Toolkit to access THOR Databases via a server. A client might be a program supplied by Daylight, one sold by a third party, or a program you write using the Daylight Toolkit. Whatever the origin of various clients, all are based on the Daylight Toolkit, the library of functions that allows programmers to write client programs (applications). Clients can vary in complexity from simple to sublime.

In spite of the variety of clients that might be available, clients typically perform the following tasks:

• Connect to one or more servers.
• Open one or more databases.
• Retrieve and/or store information in databases.
• Ask servers for information about the servers and their databases.

1.4 Current Servers and Clients Applications

Clients
Program	Client of...	Function
XVTHOR	thorserver	Read/write database access.
XVMerlin	thorserver and merlinserver	Exploratory data analysis of Merlin pools.
MCL	Merlinserver	Serial script-based Merlin client.
sthorman	THORserver and Merlinserver	THOR/Merlin server and database management.
thorchange thorcrunch thordbinfo thordbping thordelete thordestroy thordiff thorlist thorload thorlookup thorls thormake thorping thorwho	thorserver	Various serial programs for THOR management and general purpose database access.
merlindbping merlinload merlinls merlinping merlinserver merlinsmartstalk merlinwho	merlinserver	Various serial programs for Merlin management and general purpose database access.

2. THOR Server

The servers write auditing information (e.g. the names of users as they connect and the databases each user opens) to a log file; the default is to use the standard-output file. Basic installation and configuration instructions for a single server can be found in the Daylight Installation Guide.

2.1 Starting Servers

Method 1 - Run server in an existing window

This method is useful when verifying that the server is installed correctly, as it allows you to see easily see the logged messages.

Choose the window you wish to use, and type:

$DY_ROOT/bin/thorserver

Once you have a server configured correctly, we suggest you use Method 2.

Method 2: Server in "background" with logfile

In this method, the server runs in "background" mode; its log-message output goes to a file. The log file can be monitored by one or more users via the thorserv_monitor program

$DY_ROOT/bin/thorserver -THOR_LOG_FILE /tmp/thorserver.log &

You can monitor the server's log file using thorserv_monitor as follows:

$DY_ROOT/bin/thorserv_monitor /tmp/thorserver.log

If you start thorserv_monitor with no log file specified, it will use the log specified by the environment variable DY_THOR_LOG_FILE. You can monitor the server's log file from several locations simultaneously, or several users can monitor it. The thorserv_monitor program is passive, and doesn't affect the server in any way. For example, you might start the THOR server on a centralized server machine, then return to a computer in your office, from which you could "rlogin" to the server, set the DISPLAY environment variable (for X Windows) to your office-computer's display, and start a thorserv_monitor program. This would allow you to monitor the central server's activities from your office.

2.2 Stopping Servers

Killing a THOR server while users are logged in can be done but not recommended. The programs sthorman and thorwho tell you the names of users currently connected and which databases they are using. Alternately, you can examine the log file. Each server prints a list of all remaining connections when someone logs out. If the last line of the log file is "(none)", then it is safe to kill the server.

The programs dayevict and sthorman provide a means for "evicting" users, and optionally allow you to send a message to users and wait for a grace period; this gives users time in which to wrap up their business.

To kill a server started in a window (Method 1 above), select the window the server is using, then type the interrupt character (usually [Control]-C but sometimes [Delete]). Alternatively, use kill with the process id the server printed on startup.

To kill servers started in the background (Method 2 above), get the server's process id from the second line of the thorserver.log file and use it with the kill command.

Please note if you do kill a THOR server with databases still open, it will try to disconnect the clients and close the databases in a safe manner.

2.3 Auto-loading Databases

Specific databases can be opened at the time that the THOR server is started. For example, the command would start the THOR server and open the database 'database1' even when no clients are using it.

$DY_ROOT/bin/thorserver database

2.4 Multiple Servers

Normally a single instance of each server provides plenty of power for all users. However, it is sometimes necessary to run multiple THOR servers on a single computer. Please note that a database can be opened by only one server at a time and that you must have a license that allows multiple servers.

The following situations may call for multiple THOR servers:

• For different levels of security. You might have two completely separate sets of databases with different security requirements. An "open/public" server might be started using one userid, and a second "secure" server by another. Since each is run under a different user's id, each server would not be able to open the other's databases. Separate passwords files can be used (see the option DATABASE_PASSWORDS_FILE), so each server's security and access is independent of the other's.
  
  
• For exclusive access to a database. If you have a multi-user license, you can have it split across two or more services (e.g. a 7-user service and a 1-user service for an 8-user license). A THOR manager who connects to the 1-user thor server knows that no one else can connect, and thus knows that if he or she has a database open, no one else does. (Note: the "lock for exclusive access" feature is not implemented in version 4.41. Later versions of Daylight software may make it possible to lock a database without a one-user server.)
  
  
• When system limits are exceeded. The operating system imposes limits on each process; for example, many systems are configured to limit each process to 64 open files. In the case of THOR, the server starts with 4 open files, each database uses 4 files, and each client connection uses 1 file. If the 64-file limit is reached and it is not convenient to reconfigure the operating system, a second server can be a useful temporary solution.

To use multiple servers, each server must have a unique service in the /etc/services file. These THOR services must start with "thor" (e.g. "thor2", "thormgr", are permissible, "mthor" is not). We suggest something like following entries in /etc/services for two servers:

thor        5555/tcp
thor2       25557/tcp

To start a second server, use the THOR_IPC_SERVICE on the command line:

$DY_ROOT/bin/thorserver -THOR_IPC_SERVICE thor2

2.5 Sending Messages to Users

Messages can be "attached" to a THOR server, which causes the message to be sent to clients as they log in. They can also be attached to a database, in which case the message is sent to the client as it opens the database. Messages can also be "broadcast" immediately to all clients of a server, or to all clients who have a particular database open.

Each message is sent as one or more "NOTE:" entry in the normal error queue. It is up to the client program to examine the error queue and deal with the messages, so there is no guarantee that a particular client will actually deliver the message to a user. Daylight-supplied client programs (e.g. XVTHOR) attempt to post all messages as promptly as possible.

Messages should be short -- lines should be less than 70 characters wide, and the whole message should be just a few lines.

2.6 Monitoring and Maintenance

If your servers keep log files, you will occasionally (depending on activity levels) need to remove the files, as they will otherwise grow to take a significant amount of disk space. You can't simply remove a log file, since the server keeps it open for writing. (In UNIX, the space a file uses is not recovered until the last "reference" to it - a program or directory entry - is removed.) Instead, you have to stop the server, remove the file, then restart the server.

It is also a good idea to occasionally review the entries in the log file, in particular to check for repeated failed login attempts. This is often an indication that an unauthorized user is attempting to guess passwords or to otherwise violate security.

3. Merlin Server

Merlin servers must be started before any THOR or Merlin programs can access a database. The servers are "daemon" programs - they run in the background and have no direct user interaction. Once you start a server, the only way to communicate with it is through a client program.

The servers write auditing information (e.g. the names of users as they connect and the databases each user opens) to a log file; the default is to use the standard-output file. Basic installation and configuration instructions for a single server can be found in the Daylight Installation Guide.

3.1 Starting Servers

We suggest starting the server with one of the following methods.

Method 1: Run server in an existing window.

This method is useful when verifying that the server is installed correctly, as it allows you to see easily see the logged messages.

Choose the window you wish to use, and type:

$DY_ROOT/bin/merlinserver

Once you have a server configured correctly, we suggest you use Method 2.

Method 2: Server in "background" with logfile

In this method, the server runs in "background" mode; its log-message output goes to a file. The log file can be monitored by one or more users via the merserv_monitor program).

$DY_ROOT/bin/merlinserver -MERLIN_LOG_FILE /tmp/merlinserver.log &

You can monitor the server's log file using merserv_monitor as follows:

$DY_ROOT/bin/merserv_monitor/tmp/merlinserver.log

If you start merserv_monitor with no log file specified, it will use the log specified by the environment variable DY_MERLIN_LOG_FILE. You can monitor the server's log file from several locations simultaneously, or several users can monitor it. The merserv_monitor program is passive, and doesn't affect the server in any way. For example, you might start the Merlin server on a centralized server machine, then return to a computer in your office, from which you could "rlogin" to the server, set the DISPLAY environment variable (for X Windows) to your office-computer's display, and start a merserv_monitor program. This would allow you to monitor the central server's activities from your office.

3.2 Stopping Servers

Killing a Merlin server while users are logged in can be done but not recommended.

The programs dayevict and sthorman provide a means for "evicting" users, and optionally allow you send a message to users and wait for a grace period; this gives users time in which to wrap up their business.

To kill a server started in a window (Method 1 above), select the window the server is using, then type the interrupt character (usually [Control]-C but sometimes [Delete]). Alternatively, use kill with the process id the server printed on startup.

To kill servers started in the background (Method 2 above), get the server's process id from the second line of the thorserver.log file and use it with the kill command.

Please note if you do kill a Merlin server with databases still open, it will try to disconnect the clients and close the databases in a safe manner.

3.3 Auto-loading databases

Specific databases can be opened at the time that the Merlin server is started. For example, the following command would start the Merlin server and load the database 'database2' into the server's memory. This makes the database available to client programs without the need to use sthorman (described below) to load the pool.

$DY_ROOT/bin/merlinserver database2

3.4 Multi-CPU Parallel Processing

If you have a multi-CPU machine, you can improve some search times dramatically by using several CPUs to "parallelize" certain Merlin searches. The environment variable MERLIN_NPROCS indicated to the merlinserver how many CPUs to use for searches. The default is one.

The actual parallelized search is performed by "child" processes, spawned by the merlinserver; for example, parallel substructure searching is handled by a program named merlinsmartstalk.

3.5 Multiple Servers

Normally a single instance of each server provides plenty of power for all users. However, it is sometimes necessary to run multiple Merlin servers on a single computer. Unlike the THOR server, it is possible for two Merlin servers to open the same database but it is not an efficient use of the computer's memory. Please note that you must have a license that allows multiple servers.

• Merlin servers use a "time slicing" scheme to break lengthy operations into small sections, thus preventing one client from "hogging" the server. However, this time-slicing version of time sharing is not as sophisticated as the true time sharing employed by the UNIX operating system. In some cases, particularly for very large databases, performance can be improved by devoting a Merlin server to one particular database. This allows the operating system's more sophisticated time-sharing system to allocate search time to each client's requests.

• As with THOR, there may be separate security required for different databases.

To use multiple servers, each server must have a unique service in the /etc/services file. The Merlin services must start with "merlin" (e.g. "merlin2", and "merlinbig" are permissible but "bigmerlin" is not). We suggest something like following entries in /etc/services for two servers:

merlin      5556/tcp
merlin2     5558/tcp

$DY_ROOT/bin/merlinserver -MERLIN_IPC_SERVICE merlin2

3.6 Sending Messages to Users

The programs daymessage and sthorman allow you to send messages to Merlin client programs.

Messages can be "attached" to a Merlin server, which causes the message to be sent to clients as they log in. They can also be attached to a database, in which case the message is sent to the client as it opens the database. Messages can also be "broadcast" immediately to all clients of a server, or to all clients who have a particular database open.

You can "broadcast" messages to users of a Merlin database (pool), but Merlin can't reconfigure databases (it is a "read-only" server), so you must use the THOR server to attach a message to a database.

Each message is sent as one or more "NOTE:" entry in the normal error queue. It is up to the client program to examine the error queue and deal with the messages, so there is no guarantee that a particular client will actually deliver the message to a user. Daylight-supplied client programs (e.g. xvmerlin) attempt to post all messages as promptly as possible.

Messages should be short -- lines should be less than 70 characters wide, and the whole message should be just a few lines.

3.7 Monitoring and Maintenance

If your servers keep log files, you will occasionally (depending on activity levels) need to remove the files, as they will otherwise grow to take a significant amount of disk space. You can't simply remove a log file, since the server keeps it open for writing. (In UNIX, the space a file uses is not recovered until the last "reference" to it - a program or directory entry - is removed.) Instead, you have to stop the server, remove the file, then restart the server.

It is also a good idea to occasionally review the entries in the log file, in particular to check for repeated failed login attempts. This is often an indication that an unauthorized user is attempting to guess passwords or to otherwise violate security. This is discussed in more detail in the section Server and Database Security.

4. Server and Database Security

It can be difficult to secure a shared database against unauTHORized access without encumbering the users with a host of easily-forgotten passwords. This is especially true in a networked environment with a database server such as THOR: wide-area networks potentially give thousands of computers access to a THOR database, and security is a genuine problem.

There are three distinct reasons for security:

• To prevent unauTHORized access to data.
• To prevent auTHORized users from accidentally changing data.
• To keep a record of users who are accessing or changing data.

You should carefully consider these three needs separately, and keep the following three cardinal rules in mind when designing your security:

• Use secure passwords. Passwords should be at least 6 characters long, should not be in a dictionary, and should not be obvious (phone numbers, birth dates, project names, people's names, "buzzwords" in common use, etc.).

• Let people choose their own passwords. Don't assign passwords that mean nothing to the people who must remember them - the most common security violation occurs when users write down passwords because they can't remember them.

• Never make security cumbersome - it motivates people to find ways around it. You want to make it easy for people to do things right.

4.1 Security and the Operating System

Thus, in keeping with good security practices, the servers will refuse to run if started by a "powerful" user such as the super-user "root" in UNIX.

It is important that the passwords file itself be protected from tampering by unauthorized users. The standard installation procedure will, if followed, give a reasonable degree of security.

4.2 Server Security

The passwords file defines the combination of hosts and users that can connect to the server. The user names in a server's host/users list are not taken from the operating system; rather they are maintained separately by the THOR/Merlin system. Although system administrators typically choose the same user names for the THOR/Merlin system as for the UNIX or VMS system (and client programs encourage this), there is no requirement that this be the case. The THOR/Merlin system uses a completely separate password validation system from that employed by the operating system (this makes the THOR/Merlin security system independent of the operating system on which it is running).

There are three levels of security available on the THOR and Merlin servers:

• No security: If the server is started with no passwords file, any user from any host can connect to the server. The server will log the transaction, but nothing more. If this configuration is used, the only protections are the database's read, write, and executive passwords. For example, a THOR server with no security can be started as follows:

  $DY_ROOT/bin/THORserver -DATABASE_PASSWORDS_FILE ""

• Equivalent Hosts: A group of computers can be considered equivalent hosts for security purposes; that is, you form a group of mutually-trustworthy computers (perhaps those in your building but not those on the rest of the campus) and list them in THOR's passwords file. Service is automatically granted to any user on an equivalent host without further validation (i.e. users don't have to supply a username and password if they are connecting from one of the equivalent hosts). Users who are not on an equivalent host must supply a valid username and password from the passwords file. The sTHORman program provides security-modification menu selections to select the equivalent-hosts mode.

  In this mode, individual user/password entries can also be added. Such users can log in from any computer, whether or not the computer is listed as an equivalent host.

• Restricted Hosts: Similar to equivalent hosts, but opposite in effect: Only users on the list of restricted hosts are allowed to connect to the server; in addition a user must supply a valid username and password when connecting to the server. Restricted-hosts-mode and equivalent-hosts-mode are mutually exclusive. sTHORman provides security-modification menu selections to select the restricted-hosts mode.

host:crawdad
host:gator
host:cajun
user:art:1aA3h3azZaqw23DS
user:dawn:GjkO96REnK2G1Waw
user:jj:AsDF12REIO9PlLYb
user:thor:jfiIOe35W1QGh7cx
user:thorinfo:

In this example, any user from machines "crawdad", "gator", and "cajun" is allowed to log in and doesn't need to supply a password. Users "art", "dawn", and "jj" can connect from any computer if they supply a password that, when encrypted, matches the password for their usernames.

If the line host:*only* appears in the passwords file, then the hosts listed are considered restricted hosts as described above. If above line does not appear in the passwords file, then all hosts listed are considered equivalent hosts. Thus, if we added the line "host:*only*" to the above example, all users would be required to supply passwords, and only users on machines crawdad, gator, and cajun would be accepted.

It is not necessary to edit the THOR server's passwords file directly. The THOR manager program sthorman provides a way to change entries in the passwords file.

The user thorinfo is is a restricted user that is allowed to connect and ask for only "non-critical" information such as if a particular database is loaded. This allows clients to ask a server questions of interest before having to enter a password. Note that thorinfo appears in the passwords file like any other user; you can remove it or add a password if you like. Of course, if you do so, the convenience features provided by this restricted user will not be available.

4.3 Database Security

Each database has three passwords, one each for read, read/write, and executive access. These passwords are encrypted and stored in an encrypted version. Database passwords can be set or reset using sthorman or thorfilters. If you have forgotten the executive password for a database, you have no recourse but to edit the .THOR file and remove the passwords, then use sthorman or thorchange to enter new ones.

5. THOR Database Administration

A THOR Database is a collection of chemical information, stored in a set of files on a computer's disk. This section discusses the practical aspects of managing THOR databases.

The tasks of a THOR administrator include the following:

• Database design. Deciding what is to be stored in it, how it is to be represented, who will have access to the data, where it will be located.

• Datatype design. Datatypes define the meaning of data in a THOR database; the THOR manager must design and document them.

• Creating and loading databases.

• Backups. Databases need to be dumped and stored on magnetic tape or some other removable medium on a regular basis.

sthorman

An interactive, comprehensive THOR/Merlin management program.

thorfilters

A collection of single-purpose UNIX-style "filter" programs. Each program does one task, such as create a database (thormake), load a database (thorload), list available databases (thorls and merlinls), etc.

Most THOR/Merlin administrators learn the basics of THOR databases management using sthorman. As certain tasks become rote, most administrators prefer to write short "shell scripts" using the thorfilters programs. Both sthorman and thorfilters are reasonable approaches to database management. You should choose whichever is suited to your needs.

Note: SGI IRIX 6.4 and higher and Solaris 5.7 and higher can use 34 bits to access files imposing a 16GB limit on new databases. Red Hat Linux 7.1 and higher and Mac OS X are limited to 32-bit access with a maximum database size of 2GB.

5.1 Database Creation and Maintenance

Databases can be created, loaded, dumped, crunched, reconfigured, and summarized using sthorman and/or the thorfilters programs. Remember when creating and maintaining databases that database paths (filenames and directories) are relative to the server's file system, whereas THOR Datatree files (TDTs) and sthorman's log files are on the file system of the computer on which sthorman is running.

5.1.1 Creating and Loading a New Database

When creating a database, you need the following information:

database name

Name of the new database containing a complete path to the specific database files. For ease of maintenance, we recommend that you use environment variables such as $DY_THORDB to describe the path.

database type

Whether this is a regular, datatypes, or indirect-data database.

datatypes database

Name of the datatypes database in which the datatypes are defined.

indirect-data database

Name of the indirect-data database in which indirect-data expansions are defined (only required if using indirect data)

primary database size

The size of the primary-data hash table. This should be roughly equal to the number of TDTs that will ultimately be stored in the database. In general it is better to overestimate than underestimate since the hash table doesn't take much space (8 bytes per entry). See the size discussion below.

cross-reference database size

The size of the cross-reference hash table. This should be roughly equal to the number of non-SMILES identifiers in the database. See the size discussion below.

read, write, and executive passwords

The passwords for the database.

Before creating a new database, you must calculate how big to make the hash tables. The general rules is to make the hash tables the same size as the number of TDTs that are to be stored. This applies to both the primary data file and the cross-reference data file. The tdtcount program can be used to count bytes, identifiers, non-identifiers, SMILES, and total number of TDTs in a file.

Example:

tdtcount example_datatypes.tdt
9562 32 231 0 32 example_datatypes.tdt

tdtcount example.tdt
270398 1601 3333 232 232 example.tdt

• Create the datatypes database
• Create the indirect-data database
• Create the regular database
• Load the datatypes database
• Load the regular database
• Load the indirect database

thormake '$DY_THORDB/example_datatypes' 32 32
thormake -DATATYPES_DATABASE example_datatypes '$DY_THORDB/example' 232 1601
thorload example_datatypes example_datatypes.tdt
thorload example example.tdt

5.1.2 Loading New Data

New data can be loaded into an existing database using sthorman and/or the thorfilters programs. Before loading new data into a THOR database, you need to know the following information:

TDT file

The name of the data file that is to be loaded.

database

The database into which the file is to be loaded.

password

You need the write password to load new data.

error file

You can specify a file into which error messages will be reported or let error messages come to the screen.

merge data

Normally, data being loaded should be merged with the data already in the database - that is, if a TDT is entered that is already present in the database, the new and old data are merged into a single TDT. This is true even when the database is initially empty, as a particular SMILES or other identifier may occur multiple times in the input file.

replace existing data

If you choose not to merge new data with existing data, there are two possible choices: the new data should replace existing data, or new data that conflicts with existing data should not be loaded.

Note that replacing existing data requires that the database be opened with executive permission.

standardize/raw

You can choose to standardize (see the Daylight Theory Manual) new data or to load the data raw. Normally you should standardize all data that are loaded into a THOR database. The raw option should only be used when re-loading data that is already standardized (i.e. data from a previous dump of a THOR database). Loading raw data is much faster than standardizing, but you must be positive that the TDTs are already standardized.

Note that datatype-definition TDTs (those being loaded into a datatypes database) cannot be merged. When re-loading datatype definitions, you must replace the existing definitions.

major/minor report frequency

As TDTs are loaded, sthorman and thorload indicate progress by printing dots ("." - minor report) and a numeric summary (major report) on the terminal. You can choose how often these reports are made (i.e. how many TDTs between reports).

5.1.3 Dumping a Database

There are three ways to dump a database's contents using: sthorman, thorlist or thordump. Before you can dump a database, you need the following information:

     %sthorman
     Version: sthorman 4.91
 
               ....................................................
               .  STHORMAN -- Serial THOR Manager Program.        .
               .                                                  .
               .  Enter "?" for help at any prompt.               .
               .  The menu selection "." goes to top-level menu.  .
               ....................................................
 
     STHORMAN needs to be connected to a server.
 
     Enter machine the server is running on: ("dia") 
     Enter port (service) of server: ("thor") 
     Enter username: ("thor") 
     Enter server username's password: 
 
     Connected to new server.
     Current service:  dia:thor

     SERVER MENU (TOP LEVEL MENU):
     1) Connect to a new server
     2) List server's version
     3) List current users
     4) Change server security...
     5) Databases...
     6) Send message to users...
     7) Evict user...
     0) Quit
     Enter menu selection (or '?'): 2

	Current service:  dia:thor
     thorserver 4.91)

     SERVER SECURITY MENU:
     1) List users
     2) List hosts
     3) Add/Change user
     4) Add host
     5) Remove user
     6) Remove host
     7) Set exclusion mode
     0) Go back to server menu
     Enter menu selection (or '?'):

     DATABASE MENU:
      1) List known databases
      2) Change database search-path...
      3) Get database information...
      4) Create   a database...
      5) Load     a database
      6) Dump     a database
      7) Crunch   a database
      8) Change   a database's configuration...
      9) Destroy  a database
     10) Send message to users of a database...
     11) Evict users of a database...
      0) Go back to server menu
     Enter menu selection (or '?'):

     POOL MENU:
     1) List known pools
     2) Change database search-path...
     3) Pool: Load database and hold in memory
     4) Pool: Release from memory
     5) Column: Create and hold in memory
     6) Column: Release from memory
     7) Show pool info
     8) Evict users of a pool...
     0) Go back to server menu
     Enter menu selection (or '?'):

     database%dbpass@host:service:user%userpass