Manifests

This topic describes manifest files and provides an example manifest for a meta DB file. It also describes manifest searching and provides an example manifest search.

Manifest files are created with every session, meta, and packet (log) DB file and index slice directory. A manifest file is a file that describes several key pieces of information about the data to which it refers. Manifest files are written as a JSON record. Manifest files travel with the data they represent from tier to tier. If the data they represent is deleted, the manifest file is also deleted, except in the following special case. If the service has /database/config/manifest.dir configured to a valid directory, at the point when the manifest data is deleted, a copy of the manifest file is placed into the directory pointed at by manifest.dir (the directory is created if it does not exist). This enables a NetWitness Platform feature called historical manifest searching.

The intention of this process is to keep historical manifest files for years, in one location for offline querying. As you might imagine from a service running for many years, this can potentially generate hundreds of thousands of files. This should not be a concern however, as the service automatically compresses files into a single archive in order to save space when they grow too numerous. Manifest files are very small and compress well.

Example manifest ( meta-000000023.nwmdb.manifest ) for a meta DB file: 

				
{
    "filename" : "meta-000000023.nwmdb",
    "size" : 185153768,
    "fileTime" : 1403903940,
    "id1" : 150814110,
    "id2" : 159341086,
    "session1" : 4023382,
    "session2" : 4250442,
    "time1" : 1403903879,
    "time2" : 1404739851
}				
			
				
filename  = The filename for the db file the manifest represents
size      = The size in bytes of the db file
fileTime  = The time the file was created
id1       = The starting id in the file (for this example, the starting meta ID)
id2       = The last id in the file (for this example, the last meta ID)
session1  = The starting session ID of the first meta in the file
session2  = The last session ID of the last meta in the file
time1     = The POSIX time of the first "time" meta found in the file
time2     = The POSIX time of the last "time" meta found in the file

In this example manifest, the most important fields are fileTime , time1 and time2 . All three fields are written in POSIX time. time1 and time2 are the starting and stopping times of the meta recorded in the meta DB file meta-000000023.nwmdb . In particular, fileTime is always the time in which the file was created (not last modified). time1 and time2 are representative of the min and max range of the parsed data within the meta DB file. When doing historical searches by time, time1 and time2 are preferred over fileTime , when they are present. Manifest files for the other databases and index contain some different fields, but all have enough information to perform time based queries.

Search Historical Manifests

When manifests are collected in the directory pointed to by manifest.dir , it is assumed that the data they refer to was copied to the Cold tier and eventually backed up to offline storage. Because the historical manifests are still accessible by the service, this allows time-based queries to be performed on offline data, in order to determine what data needs to be restored for a given time range.

You can search manifests using the /database manifest command: 

				
manifest: If a manifest directory is defined, it will allow operations on the manifest files (such as a time based query) for database files in cold storage.
security.roles: database.manage
parameters:
   op - <string, optional, {enum-one:query|compress}> The operation to perform (defaults to query)
   time1 - <date-time, optional> The beginning time (UTC) for matching offline database files
   time2 - <date-time, optional> The ending time (UTC) for matching offline database files
   timeFormat - <string, optional, {enum-one:posix|simple}> Specify the time format that is returned (posix, simple), default is posix

Example search:

/database manifest time1="2014-04-20 11:00:00" time2="2014-04-11 11:20:00" timeFormat=simple

The search returns all manifests that match the query:

				
[ filename=meta-000001691.nwmdb size=4843826176 fileTime="2014-Apr-20 11:06:34" id1=301555027452 id2=301733101896 session1=15352020201 session2=15361024200 time1="2014-Apr-20 11:05:34" time2="2014-Apr-20 11:16:34" compression=gzip ]
[ filename=session-000001865.nwsdb size=268439552 fileTime="2014-Apr-20 11:06:35" id1=14674145801 id2=14682041000 metaId1=288217522208 metaId2=288370660984 packetId1=11733872441 packetId2=11741745303 ]
[ filename=session-000001866.nwsdb size=268439552 fileTime="2014-Apr-20 11:18:31" id1=14682041001 id2=14689936200 metaId1=288370660985 metaId2=288520616949 packetId1=11741745304 packetId2=11749618589 ]

The returned results can be used to correlate which files should be restored from backup for the given time range. For versions 10.4 and later, a service called Workbench can be used to take the restored files and provide a query interface over the restored data using one or more collections.

Setup of the Workbench service is beyond the scope of this document. For more information, see "Configure Data Backup and Restore" in the Archiver Configuration Guide .