Note to developers/maintainers: The documentation for the individual sam_admin commands is also maintained as a part of the individual autodest commands. Any changes made in this document should also be made there as well (and vise-versa). Sorry for the redundancy but haven't found a way around it yet.
$ samadmin autodest get mapfile --mapfile=myMap.py # obtain the master $ samadmin autodest synch mapfile --mapfile=myMap.py # synch reality with specified mapfile $ samadmin autodest load mapfile --mapfile=myMap.py [--verify=all] # # OBSOLETE and very very slow: # $ samadmin autodest verify mapfile --mapfile=myMap.py # verify a mapfile # $ samadmin autodest synch mapfile --mapfile=myMap.py # bring reality into synch # $ samadmin autodest load mapfile --mapfile=myMap.py # synch AND upload map #
Before using any of the following commands, you should:
$ setup [-q dev|int|prd] sam
The autodest maintenance commands all use the following syntax:
$ samadmin autodest [subcommand] [--options] [-flags]and all adhere to the following conventions/rules:
The following procedure should generally be followed when updating or maintaining the map:
1. Obtain a copy of the Master Map from the database: | $ samadmin autodest get master mapfile --mapfile=myMapFile.py |
2. Edit the mapfile | $ xemacs myMapFile.py # or your personal favorite editor |
3. Verify the mapfile: | $ samadmin autodest verify mapfile --mapfile=myMapFile.py [--verify=all] |
4. Synchronize reality with your new mapfile: | $ samadmin autodest synch mapfile --mapfile=myMapFile.py [--synch=all] |
5. Load the new mapfile: | $ samadmin autodest load mapfile --mapfile=myMapFile.py [--verify=all] |
Other commands that may be useful to know are:
6. Verify the Master Map: | $ samadmin autodest verify master mapfile [options] |
7. Renumber mapfile: | $ samadmin autodest renumber mapfile --mapfile=myMapFile.py [options] |
8. Create a new pnfs location: | $ samadmin autodest create location --path=/pnfs/new/path [options] |
9. Modify an existing pnfs location: | $ samadmin autodest modify location --path=/pnfs/path [options] |
10. Verify a location: | $ samadmin autodest verify location --path=/pnfs/path [options] |
11. Test an autodest string: | $ samadmin autodest teststring --path=stringToTest [options] |
$ samadmin autodest get master mapfile --mapfile=<myMapFile.py>The database contains the master of the autodestination mappings. Whenever you wish to add new entries, you should first get the master map. Make your modifications to this file.
Items to Note:
The mapfile will look similar to:
destList = [ { # map entry number 0: 'pathPattern' : '(/pnfs/sam/mammoth/copy1/monte_carlo/)([^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+)', 'destinationPath' : '/pnfs/sam/mammoth/mcc99_2/in2p3', 'library' : 'sammam', 'file_family' : 'in2p3', 'file_family_wrapper' : 'cpio_odc', 'storage_group' : 'D0', 'file_family_width' : '1', 'permissions' : 'rwxrwxr-x', }, { # map entry number 1: 'pathPattern' : '(/pnfs/sam/mammoth/copy2/monte_carlo/)([^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+)', 'destinationPath' : '/pnfs/sam/mammoth/dev/mcc99_2/in2p3', 'library' : 'sammam', 'file_family' : 'in2p3_dev', 'file_family_wrapper' : 'cpio_odc', 'storage_group' : 'D0', 'file_family_width' : '1', 'permissions' : 'rwxrwxr-x', }, ]Items to Note::
$ samadmin autodest verify mapfile --mapfile=<myMapFile.py> [--verify=all]After you have made your changes to the mapfile, you should verify it. This will compare the entries in your mapfile to "reality" and report on any differences. By default, only NEW entries will be verified; entries which already exist in the master map will not be re-verified.
Note, to see the "real" values of pnfs tags, you can
$ setup encp -q d0en # or appropriate qualifier $ cd /pnfs/sam/path/of/interest $ enstore pnfs --tags
A typical report would look like:
Verifying map entry number 0 (path = /pnfs/sam/mammoth/mcc99_2/in2p3): SUMMARY: Invalid pathPattern in map entry; Attribute mismatch pathPattern: INV. EXPR. mapfile = (/pnfs/sam/mammoth/copy1/monte_carlo/)[^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+) actual = <N/A> library: MISMATCH mapfile = samnull actual = sammam file_family: NEW TAG VALUE mapfile = in2p3_def actual = in2p3 permissions: MISMATCH mapfile = rwxrwx--- actual = rwxrwxr-x Verifying map entry number 1 (path = /pnfs/sam/mammoth/dev/mcc99_2/in2p3_dev): SUMMARY: Path does not exist; Path is not a valid SAM location; Attribute mismatch destinationPath: NO SUCH PATH mapfile = /pnfs/sam/mammoth/dev/mcc99_2/in2p3_dev actual = DOES NOT EXIST library: mapfile = samnull actual = <unknown> file_family: mapfile = in2p3_dev actual = <unknown> file_family_wrapper: NEW TAG VALUE mapfile = cpio_odx actual = <unknown> storage_group: mapfile = D0 actual = <unknown> file_family_width: mapfile = 1 actual = <unknown> permissions: mapfile = rwxr-xr-x actual = <unknown> Verifying map entry number 2 (path = /pnfs/sam/mammoth/dev/mcc99_2/in2p3): SUMMARY: Attribute mismatch library: MISMATCH mapfile = samnull actual = sammam Verifying map entry number 3 (path = /pnfs/sam/NULL/test/mcc99): SUMMARY: Verified. Verifying map entry number 4 (path = d0mino.fnal.gov:/sam/cache22/bogus/boo): SUMMARY: Path is not a valid SAM location pathPattern: Ok. mapfile = (/pnfs/lauri)(.*) actual = <n/a> destinationPath: NOT IN SAM mapfile = d0mino.fnal.gov:/sam/cache22/bogus/boo actual = <not a /pnfs path>
How to Read the Verification Report
The SUMMARY line contains a summary of problems found with each map entry.
If there are inconsistencies, the summary may contain one or more of the following:
Summary statement | Interpretation |
---|---|
Path does not exist | The specified destinationPath does not exist (or cannot be stat'ed on the current machine). If destinationPath is in /pnfs space (i.e., begins with "/pnfs", it will be created if you synchronize or load the mapfile; or you may create the directory manually with the create location command. (The destinationPath directory will not be created for you if it is not in /pnfs space). |
Path is not a valid SAM location | The specified destinationPath is not a recognized location in the SAM database. If destinationPath is in /pnfs space, then it will be entered into the SAM database as a valid tape location if you synchronize or load the mapfile; or you may create the tape location manually with the create location command. (The destinationPath will not be registered in the SAM database for you if it is not in /pnfs space). |
Missing required element(s) in map entry | The map entry is missing one or more required elements. This must be corrected before you can successfully synchronize or load the mapfile. (Not applicable when destinationPath is not in /pnfs space). |
Invalid pathPattern in map entry | The pathPattern is not a valid regular expression; most likely, it is missing some required punctuation, etc. This must be corrected before you can successfully synchronize or load the mapfile |
Attribute mismatch | The attribute specified in the mapfile does not match the actual value. Reality will be altered to match the mapfile if you synchronize or load the map. (Not applicable when destinationPath is not in /pnfs space). |
Following the SUMMARY are the details of any inconsistencies found, in the format:
entityName: ERROR FLAG mapfile = mapFile-value actual = real-valuewhere:
Detail statement | Interpretation | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
entityName | pathPattern, destinationPath, file_family, etc. | ||||||||||||
ERROR FLAG | Any of the following:
|
||||||||||||
mapFile-value | Value specified in the mapfile | ||||||||||||
real-value | Value found by actual examination of the destinationPath in /pnfs space |
Items to Note::
$ samadmin autodest synch mapfile --mapfile=<myMapFile.py> [] [--synch=all]
You have made changes to the mapfile. You have verified the new mapfile, and are satisfied that any differences between the mapfile and "reality" should be corrected, using the mapfile as the "master" information source. You are now ready to alter pnfs space (and, if necessary, the sam data_storage_locations) to match the new mapfile.
The synch mapfile command will read through the mapfile and create any new pnfs directories that are listed but do not yet exist; modify pnfs directory tags and permissions so that they match what is in the map file; ensure that all destinationPath entries are valid locations in the SAM database. By default, only new entries will be synchronized; entries which already exist in the master map will not be re-synched.
Items to Note::
$ samadmin autodest load mapfile --mapfile=<myMapFile.py> [] [--nocommit] [--verify=all]
When you load the new mapfile, any new entries will be verified and synched before the records are uploaded. If, for some reason, the autodest script cannot make reality match the new mapfile, it will not be uploaded to the database.
Items to Note::
$ samadmin autodest verify master map
This command will compare the database map to reality and report on any inconsistencies.
$ samadmin autodest renumber mapfile --mapfile=<myMapFile.py>
This command will read a file and rewrite it, changing only the comments indicating
map entry number. It is provided to make it easier to determine which entries have
problems when running samadmin autodest verify mapfile.
$ samadmin autodest create location --path=/pnfs/new/path \ [--permissions=<perm>] [--library=<lib>] \ [--file_family=<ffam>] [--file_family_wrapper=<ffw>] \ [--storage_group=<sg>] [--file_family_width=<ffwid>] \ []
This command will allow you to manually create a new pnfs directory (and enter it into the sam database if necessary) with the specified attributes.
Items to Note::
$ samadmin autodest modify location --path=/pnfs/new/path \ [--permissions=<perm>] [--library=<lib>] \ [--file_family=<ffam>] [--file_family_wrapper=<ffw>] \ [--storage_group=<sg>] [--file_family_width=<ffwid>] \ []
This command will modify the attributes of an existing pnfs directory.
Items to Note::
$ samadmin autodest verify location --path=/pnfs/path
This command will verify that the specified location is a valid SAM location, and that the directory exists. It will print out the current attributes. A typical report would look like:
$ samadmin autodest verify --path=/pnfs/sam/NULL /pnfs/sam/NULL is a valid SAM location. Location /pnfs/sam/NULL exists with properties: file_family = jon file_family_width = 1 file_family_wrapper = cpio_odc library = samnull permissions = rwxrwxr-x storage_group = D0Items to Note:
$ samadmin autodest teststring --path=/path/to/test \ [--pattern=<testPattern> | --mapfile=<myMapFile.py>]
This command will compare a given path and determine if it matches the pattern provided, or any of the patterns in the specified mapfile. If neither a pattern nor a mapfile is provided on the command line, the path will be compared to the master mapfile.
A sample command looks like the following:
$ samadmin autodest teststring \ --path=/pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3 Testpath /pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3 matches map entry number 4: (/pnfs/sam/mammoth/copy1/monte_carlo/)([^/]+)(/mcpc_uta/)([^/]+)(/)([^/]+) would be mapped to path /pnfs/sam/mammoth/mcc99_2/uta Testpath /pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3 matches map entry number 22: /.*/ would be mapped to path /pnfs/sam/NULL
There is no automated tool at this time. Information is available on the web at http://d0ensrv2.fnal.gov/enstore/pnfsRate.html
The relevant info is:
Where in the example above the db2 database shows 600+Mb used.
The individual files can grow up to 2 GB. We should switch to a new database when the file systems gets between 1.2 - 1.7 Gb in size.
DB Entity Description PNFS_PATTERN a regular expression for string matching PATH_PRIORITY a number representing the order of patterns to be matched (first match 'wins'); this number is generated automatically when the database table is loaded AUTO_DEST_FAMILY encp file_family for this pnfs directory LOCATION_ID an index into the DATA_STORAGE_LOCATIONS table (e.g., a mapping to the file name) LIBRARY encp library for this pnfs directory FILE_FAMILY_WRAPPER encp file_family_wrapper for this pnfs directory STORAGE_GROUP encp storage_group for this pnfs directory FILE_FAMILY_WIDTH encp file_family_width for this pnfs directory PERMISSIONS the unix file permissions for this pnfs directory
============================================================================= Project : SAM Package : sam_admin_pyapi $Id: autodest.html,v 1.1 2004/03/24 19:53:07 lauri Exp $ This work is part of a development project, called SAM, which consists of a number of coordinated packages each named sam_xxxx . Notice of authorship, copyright status, and terms and conditions, should the software eventually become available for use outside Fermilab, can be found in the README and LICENCE files in the top level directory of the main sam package. ==============================================================================
Date Author Description
2004/03/24
llc
cleanup, brushup, for sam_admin_pyapi re-implementation.
2003/06/06
weigand
Added 'audit directories' command.
Added 'Contents' section
Added 'Utilities for Auditing the Auto-Destination PNFS Locations' section