FASTFIND LINKS Document Organization Product Version Getting Help Table of Contents
MK-96ARC005-10
Copyright 20072011 Hitachi Data Systems Corporation, ALL RIGHTS RESERVED No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for any purpose without the express written permission of Hitachi Data Systems Corporation (hereinafter referred to as Hitachi Data Systems). Hitachi Data Systems reserves the right to make changes to this document at any time without notice and assumes no responsibility for its use. This document contains the most current information available at the time of publication. When new and/or revised information becomes available, this entire document will be updated and distributed to all registered users. Some of the features described in this document may not be currently available. Refer to the most recent product announcement or contact your local Hitachi Data Systems sales office for information about feature and product availability. Notice: Hitachi Data Systems products and services can be ordered only under the terms and conditions of the applicable Hitachi Data Systems agreement(s). The use of Hitachi Data Systems products is governed by the terms of your agreement(s) with Hitachi Data Systems. By using this software, you agree that you are responsible for: a) Acquiring the relevant consents as may be required under local privacy laws or otherwise from employees and other individuals to access relevant data; and b) Ensuring that data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws. Hitachi is a registered trademark of Hitachi, Ltd. in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd. in the United States and other countries. All other trademarks, service marks, and company names are properties of their respective owners.
Contents
Preface........................................................................................................vii
Intended audience . . . . Product version . . . . . . Document organization . Syntax notation . . . . . . Related documents. . . . Getting help. . . . . . . . . Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... ... ... ... ... ... ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... .... .... .... .... .... .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii . vii . viii . .ix ..x . xii . xii
Data movement workflow . . . . . . . . . . . . . . Task 1: Identifying the data . . . . . . . . . Task 2: Deciding where to put the data . Task 3: Copying or moving the data . . . Using scripts . . . . . . . . . . . . . . . . . . . . Workflow examples . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
iv
Metadata properties . . . . . . . . . . . . . . . . . . . . . . HCP namespace metadata properties . . . . . . . . Default namespace metadata properties . . . . . . Namespace directory structure. . . . . . . . . . . . . . . New directories . . . . . . . . . . . . . . . . . . . . . . . . . Optimizing performance . . . . . . . . . . . . . . . . . . . Retrying on failure . . . . . . . . . . . . . . . . . . . . . . . arcmv errors . . . . . . . . . . . . . . . . . . . . . . . . . . . arcmv parameter interactions . . . . . . . . . . . . . . . Overwriting using the same source and destination Custom metadata considerations . . . . . . . . . . . . . arcmv examples . . . . . . . . . . . . . . . . . . . . . . . . . . . .
....... ....... ....... ....... ....... ....... ....... ....... ....... directory ....... .......
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
.5-22 .5-23 .5-23 .5-24 .5-24 .5-25 .5-26 .5-26 .5-26 .5-27 .5-27 .5-28
Glossary Index
vi
Preface
This book contains all the information you need to use the set of command-line client tools distributed with Hitachi Content Platform (HCP). For information on installing the client tools, see Managing a Tenant and its Namespaces or Managing the Default Tenant and Namespace. Note: Throughout this book, the word Unix is used to represent all UNIXlike operating systems (such as UNIX itself or Linux), except where Linux is specifically required.
Intended audience
This book is intended for IT professionals who are responsible for storing corporate data in an HCP namespace. It assumes you:
Are familiar with the file systems and namespace access protocols in
use at your site
Have experience using command-line interfaces Have a basic understanding of HCP technology
Product version
This document applies to release 4.1 of HCP.
vii
Document organization
Document organization
This book contains seven chapters and a glossary.
Chapter/Appendix
Chapter 1, Introduction to Hitachi Content Platform Chapter 2, Introduction to the HCP client tools Chapter 3, Working with the HCP client tools
Description
Introduces basic HCP concepts that you need to understand in order to successfully use the HCP client tools Introduces the HCP client tools and explains conventions and examples used in the rest of the book Explains how the HCP client tools fit into the data storage workflow and discusses other tools that provide the same functionality Presents an overview, syntax, command considerations, and examples of the arcfind tool Presents an overview, syntax, command considerations, and examples of the arcmv tool Presents an overview, syntax, command considerations, and examples of the arcmkdir tool Describes two real-life use cases in which customers are successfully using the HCP client tools
Chapter 6, Creating directories (arcmkdir) Chapter 7, HCP client tools use cases
Appendix, Installing the HCP client tools Describes how to install compiled HCP client tool files and to client tool source code
viii
Syntax notation
Syntax notation
The table below describes the conventions for HCP client tool command syntax.
Notation boldface Meaning
Type exactly as it appears in the syntax (if the context is case insensitive, you can vary the case of the letters you type) Replace with a value of the indicated type Vertical bar Choose one of the elements on either side of the bar, but not both Square brackets Include none, one, or more of the elements between the brackets Parentheses Include exactly one of the elements between the parentheses Replace with the combination of the directory path and name of a file Replace with a directory path with no file name Ellipsis Optionally, repeat the preceding parameter as many times as needed
Example
This book shows: arcfind You enter: arcfind
italics |
This book shows: directory-path You enter: HR/Benefits/Health This book shows: arcfind -h|--help You enter: arcfind -h or: arcfind --help This book shows:
[ ]
( )
-file-spec
-path ...
This book shows: input-directory-path You enter: HR/Benefits This book shows:
ix
Related documents
Related documents
The following documents contain additional information about Hitachi Content Platform:
Related documents
Using the Default Namespace This book describes the file system
HCP uses to present the contents of the default namespace. It provides instructions for accessing the namespace by using the HCP-supported protocols for the purpose of storing, retrieving, and deleting objects, as well as changing object metadata such as retention and permissions.
Using HCP Data Migrator This book contains the information you
need to install and use the HCP Data Migrator (HCP-DM) utility distributed with HCP. This utility enables you to copy data between local file systems, HCP namespaces, and earlier HCAP archives. It also supports bulk delete operations. The book describes both the interactive window-based interface and the set of command-line tools included in HCP-DM.
xi
Getting help
Getting help
The Hitachi Data Systems customer support staff is available 24 hours a day, seven days a week. If you need technical support, please call:
United States: (800) 446-0744 Outside the United States: (858) 547-4526
Note: If you purchased HCP from a third party, please contact your authorized service provider.
Comments
Please send us your comments on this document: hcp.documentation.feedback@hds.com Include the document title, number, and revision, and refer to specific sections and paragraphs whenever possible. Thank you! (All comments become the property of Hitachi Data Systems.)
xii
1
Introduction to Hitachi Content Platform
Hitachi Content Platform (HCP) is a distributed storage system designed to support large, growing repositories of fixed-content data. An HCP repository is partitioned into namespaces, each of which stores both data and metadata about that data. This chapter provides a brief overview of some of the concepts you need to understand in order to successfully use the HCP client tools.
11
12
Metadata
Metadata
Data has metadata, both before you store it in a namespace and once its in the namespace. Metadata is information about data. Outside HCP, files have POSIX metadata such as creation and access times. In HCP, files have system metadata and custom metadata. System metadata consists of HCP-specific information such as retention and shred settings and, for default namespaces only, POSIX metadata. Custom metadata consists of user-defined properties. By default, when you add a file to an HCP namespace, it inherits system metadata settings from the namespace configuration. When you add a file to the default namespace, it inherits system metadata settings from the directory you store it in. For more information on metadata, see Using a Namespace or Using the Default Namespace.
13
Metadata
14
2
Introduction to the HCP client tools
The HCP client tools are a set of command-line tools provided with Hitachi Content Platform. These tools allow you to store and manage files in namespaces. This chapter describes how the client tools fit into HCP. It also explains the command conventions and examples used in this book.
Introduction to the HCP client tools Using the HCP Client Tools
21
The arcfind tool generates a list of the files you want to copy or
move. Files can be in a local or remote system or a default namespace. arcfind cannot find files in HCP namespaces.
The arcmv tool copies or moves the files in a list from a location on a
client computer or from a namespace to another location on a client computer or namespace. arcmv is a high-performance, multithreaded tool designed to handle large numbers of files at a time.
22
Introduction to the HCP client tools Using the HCP Client Tools
The big picture The figure below shows the relationship between the HCP client tools and HCP.
HCP
Some keyword parameters can occur multiple times in a single command line. When including a recurring keyword parameter in a command line, you need to include the keyword in each occurrence.
Introduction to the HCP client tools Using the HCP Client Tools
23
An argument is a value you supply that tells the tool what to work on,
and is not preceded by a keyword. In the HCP client tool commands, only arcfind and arcmkdir take arguments, and in both cases, the arguments are one or more directory paths separated by spaces; for example:
HR/Benefits HR/Contacts
The arcmv command does not take explicit arguments. Instead, it works on a list of files contained in the file specified by the --infile keyword parameter, or input from stdin.
Keyword prefixes
Each keyword in an HCP client tool command line is preceded by a single hyphen (-) if it consists of a single letter or two hyphens (--) if it consists of two or more letters. You can concatenate single-letter keyword parameters with each other as long as none before the last one take user-supplied values. When you do this, you drop the hyphens between the letters. For example, in an arcfind command, you can concatenate the -v, -L, and -e parameters like this:
-vLe ..\PDFfiles-hr-2010-10-09.log
Tip: When concatenating single-letter options, be sure to precede the string with a single hyphen. If you use a double hyphen, the tool treats the entire string as one keyword.
Command-line structure
In an HCP client tool command line, keyword parameters always come before any arguments. The keyword parameters themselves, however, can be in any order. Most keyword parameters are optional, and you can include none, one, or more of them.
arcmv and arcmkdir each have one required keyword parameter. These
tools dont execute if you omit that parameter from the command line. You use spaces to separate the parameters in HCP client tool commands.
Case sensitivity
The keywords in HCP client tool command lines are case sensitive in both Windows and Unix. The command names, however, and file and directory specifications are case sensitive in Unix, as expected, but not in Windows.
24
Introduction to the HCP client tools Using the HCP Client Tools
Common parameters
Some command parameters are available in more than one tool. All three tools, for example, support the --help and --version parameters:
--help causes the tool to display its syntax rules. --version causes the tool to display its version number.
If you issue an HCP client tool command without any parameters, the tool displays both its help and version number. Likewise, if the command line contains a syntax error or the --help parameter anywhere in it, the tool displays both its help and version number and doesnt execute, even if other parameters are valid. Additional common keyword parameters deal with logging. They are:
--quiet (in arcfind and arcmv) --verbose (in all three tools) --logfile (in arcfind and arcmv)
To specify a plus sign (+) in a URL, use %2b or %2B. To specify a percent sign (%) in a URL, use %25.
Introduction to the HCP client tools Using the HCP Client Tools
25
When a file specification or directory path refers to a local or remote file system, use the conventions specific to the platform youre working from. For example, in Windows, use the backslash (\) as the separator in directory paths; in Unix, use the forward slash (/). When the path identifies an HTTP or WebDAV location, use only forward slashes. Note: HTTP automatically converts backslashes in Windows file specifications and directory paths to forward slashes before sending them to HCP.
26
Introduction to the HCP client tools Using the HCP Client Tools
3
Working with the HCP client tools
You can use the HCP client tools as well as tools that are native to your operating system or access protocol to copy or move data from a client to HCP. Whichever tools you choose, however, the workflow remains the same. You need to find the data, decide where it goes, and put it there. This chapter discusses the kinds of tools you need and where they fit into the workflow of copying or moving data to an HCP namespace.
Working with the HCP client tools Using the HCP Client Tools
31
It has both GUI and command-line interfaces. It supports bulk deletes in the GUI and provides an hcpdm delete
command that can do batch deletes.
The GUI interface supports drag-and-drop copying within the GUI and
From Windows Explorer to the GUI.
The GUI lets you monitor the progress of copy and delete operations.
The displayed information includes metrics such as files transferred per second, succeeded and failed copies, and elapsed time. This information can be particularly helpful in tracking large migration jobs.
The GUI lets you interrupt, save, and resume jobs. You can resume
jobs that have been intentionally interrupted or that could not complete due to errors.
The GUI lets you lets you dynamically configure the load on the HCP
system and lets you configure a reduced load for specific times of day. For information on some of the limitations that HCP-DM has compared to the individual client tool commands, see Selecting the tool to use on page 3-3.
32
Working with the HCP client tools Using the HCP Client Tools
application-specific database queries, have more limited capabilities. arcmv arcmv, which provides both copy and move capabilities, is multithreaded, resulting in a high rate of throughput. arcmv supports multiple data transfer protocols with the same command syntax. With arcmv, you can map source locations to target locations with different names and paths, thereby allowing the HCP namespace to have a different directory structure from that of any of the data sources. Additionally, when overwriting a file, arcmv first deletes the existing file, then writes the new one, in accordance with WORM semantics. Because HCP has permission and retention settings that affect file deletion, files you cannot delete are protected from being overwritten. HCP-DM provides an hcpdm copy command that is similar to arcmv but provides a different and more limited set of parameters and features. Here are some of the more significant limitations on hcpdm copy:
It does not include an overwrite capability. It cannot delete files from the source after they have been copied, but
you can use the hcpdm delete command to do so.
It cannot require the source files to have custom metadata files and can
apply only a single custom metadata file to all files copied from the local file system.
Working with the HCP client tools Using the HCP Client Tools
33
It cannot preserve the source atime and mtime values for files copied to
the default namespace. The values for copies are always the time they are added to the namespace. To decide which command is more appropriate for your purposes, you should compare the command descriptions in detail. Other tools you can use to copy or move a group of files, such as the single-operation copy and move commands and curl, which has a protocoldependent syntax, have more limited capabilities. arcmkdir
arcmkdir is a multiplatform, multiprotocol tool for creating empty directories. You can use it to create directories on local and remote file systems and in a default namespace. HCP-DM does not provide an equivalent command, but you can use the HCP-DM GUI to create empty directories. The Windows and Unix mkdir commands are limited to local and remote file systems.
Creation date or last modification date Location in the file system hierarchy
34
Working with the HCP client tools Using the HCP Client Tools
Partial file name or file name extension File size Person who created the file Organization that owns the file Time period covered by the file (for example, first quarter or yesterday)
Which tool you use for this task may depend partially on the types of search criteria the tool supports. For other factors that can affect your choice of tool, see arcfind on page 3-3. For considerations regarding lists of files generated by arcfind for use with the arcmv tool, see arcfind output as arcmv input on page 4-11.
Working with the HCP client tools Using the HCP Client Tools
35
Which protocols are available and how much traffic theyre currently
experiencing.
How much data youre copying or moving. The faster the protocol, the
better it can handle large amounts of data without affecting other HCP activities.
Using scripts
Scripts enable you to put multiple commands together so you dont need to enter them individually. Putting copy and move operations into a script has these advantages:
You can use a Windows scheduled task or a Unix cron job to run the
script automatically at particular times (for example, at 1:00 a.m. every night). Tip: When large amounts of data are being copied or moved, run the script when the demand for network resources is low.
You can use variables such as SYSTEMROOT and DATE to make the
script more reusable.
36
Working with the HCP client tools Using the HCP Client Tools
Workflow examples
You can set the base directory so relative paths in the script work every
time it runs.
You can ensure that the copy and move commands are well-formed
each time theyre invoked.
Workflow examples
The sample scripts below the first for Windows, the second for Unix show the workflow for backing up files from a client computer to an HCP namespace. The scripts use arcfind to locate the files to be copied and pipe the resulting list of files directly into arcmv. You can schedule scripts like these to create daily backups. Windows example Heres the sample script:
set DATE=2010-10-24 arcfind --ctime "+%DATE% 00:05:00" --minsize 1 -e findlog.txt ^ -v \Benefits | arcmv --src "C:" --dst http://finance.europe.hcp.^ example.com/rest --dstroot "Backup-%DATE%" -v -l 800 ^ --logfile Backup-log.%DATE%.txt
Heres what this script does: 1. Sets the DATE variable to 2010-10-24 2. Invokes arcfind, which:
a. Searches in C:\Benefits for files with metadata that was last modified
after 12:05 a.m. on 2010-10-24 and that contain at least one byte of data
b. Writes session and file statistics to a file named findlog.txt in the
Note: The vertical bar (|) is the symbol for piping. The caret (^) is the line continuation character. 3. Invokes arcmv, which:
a. Looks on the C: drive of the client computer for the files being piped
in.
Working with the HCP client tools Using the HCP Client Tools
37
Workflow examples b. Writes the files it finds to HCP starting at this base location: http://finance.europe.hcp.example.com/rest/Backup-2010-10-24
The remainder of the file paths consist of the source file paths, starting with \Benefits.
c. Writes file statistics to stderr and file statistics, session statistics, all
informational, warning, and error messages, with some additional processing information to a file named Backup-log.2010-10-24.txt in the current working directory Unix example Heres the sample script:
#!/bin/sh DATE='date +%F' FINDLOG=findlog arcfind --ctime "+$DATE 00:05:00" --minsize 1 -e $FINDLOG /benefits \ | arcmv --dst http://finance.europe.hcp.example.com/rest \ --dstroot "Backup-$DATE" -v -l 800 --logfile Backup-log.$DATE
Heres what this script does: 1. Sets the DATE variable to the current date with the format yyyy-mmdd 2. Sets the FINDLOG variable to findlog 3. Invokes arcfind, which:
a. Searches /benefits for files with metadata that was last modified
after 12:05 a.m. on the current date and that contain at least one byte of data
b. Writes session and file statistics to a file named findlog in the
Note: The vertical bar (|) is the symbol for piping. The backslash (\) is the line continuation character.
38
Working with the HCP client tools Using the HCP Client Tools
Workflow examples
DATE is the value set in step 1. The remainder of the file paths consist of the source file paths, starting with /benefits.
c. Writes file statistics to stderr and file statistics, session statistics,
all informational, warning, and error messages, with some additional processing information to a file named Backup-log.valueof-the-DATE-variable in the current working directory, where DATE is the value set in step 1
Working with the HCP client tools Using the HCP Client Tools
39
Workflow examples
310
Working with the HCP client tools Using the HCP Client Tools
4
Finding files (arcfind)
The arcfind tool lets you generate lists of files or directories that you can use as input to other tools, such as arcmv. In the arcfind command line, you specify criteria that arcfind uses to select the items youre interested in. arcfind searches for these items only within the directory trees you identify in the command line. This chapter presents instructions for using arcfind. Note: You cannot use arcfind to find files in HCP namespaces.
41
About arcfind
About arcfind
The arcfind tool searches for files or directories in one or more directory trees. In the command line to invoke the tool, you specify the top-level directory of each tree you want to search. You can identify the directory either with an absolute path or the path relative to the current working directory. You cannot use arcfind to search a namespace with HTTP or WebDAV. To use arcfind, you need to map or mount the namespace so it appears to be part of the native file system. By default, arcfind returns a list of files, but you can use a command-line parameter to get a list of directories instead. Tip: You can pipe the output from arcfind directly into the arcmv tool. As compared with creating and saving a file, piping makes more efficient use of the resources on your computer. It also makes the arcfind and arcmv commands simpler and saves you keystrokes. Important: If you plan to pipe the output from arcfind directly into the arcmv tool, be sure to read arcfind output below, arcfind output as arcmv input on page 4-11, and Source and target file locations on page 5-17 before you construct each command.
directories that match the criteria you supply using command-line parameters. Using these parameters, you can filter files or directories based on time attributes. Additionally, you can filter files based on their size or by comparing their names with one or more text strings you specify. You can also use command-line parameters to restrict the search to only a portion of each directory tree. You can do this in two ways:
42
About arcfind
arcfind output
Output from arcfind is a list of files or directories. You can specify whether the list is written to a file or displayed on-screen. Within the list, the path shown for each item starts with the path of the top-level directory exactly as it appears in the command line. For example, if you specify the top-level directory as test/test_1, the resulting list of files would look something like this:
test/test_1/file_a test/test_1/file_b test/test_1/test_2/file_c test/test_1/test_2/file_d test/test_1/test_X test/test_1/Z_test/file_k test/test_1/Z_test/test_7/temp_file
Tip: In Windows, use WordPad rather than Notepad to open arcfind output files. In WordPad, each listed file appears in a separate line. In Notepad, the files are listed in a single line.
arcfind log
In addition to the output list, arcfind generates a log that contains information about its processing. You can specify whether the log is written to a file or displayed on-screen, as well as what type of information it contains. Tip: In Windows, use WordPad rather than Notepad to open arcfind log files. In WordPad, each listed file appears in a separate line. In Notepad, the files are listed in a single line. Statistics in the arcfind log The statistics in the arcfind log are labeled for easy search and script postprocessing. The table below describes these labels.
Label
Directories processed pruned max depth
Description
The number of directories arcfind encountered as it walked down the directory trees The number of those directories arcfind did not process because of --prune parameters in the arcfind command line The number of levels arcfind walked down the directory trees
43
About arcfind
(Continued)
Label
max passing entries Files processed passed filtered others
Description
The highest number of files arcfind selected from a single directory The number of files arcfind encountered as it walked down the directory trees The number of encountered files that met the criteria specified in the arcfind command line The number of encountered files arcfind filtered out based on the criteria specified in the command line The number of encountered files arcfind did not process for any other reason (for example, because what appeared to be a data file was actually a socket or device file) The total size, in KiB, of the files that passed The average size, in KiB, of the files that passed See Distribution of file sizes below
Distribution of file sizes As part of its statistics, an arcfind log includes the distribution of the selected files by size. The distribution ranges are:
< 1KiB 1KiB - 10KiB 10KiB - 100KiB 100KiB - MiB 1MiB - 10MiB 10MiB - 100MiB 100MiB - 1GiB 1GiB - 10GiB 10GiB - 100GiB >= 100GiB
In these ranges:
KiB = 1024 bytes MiB = 10242 bytes GiB = 10243 bytes
44
arcfind syntax
arcfind syntax
Heres the syntax for the arcfind command:
arcfind (-h|--help) |--version |([--dirs] [-L|--link] [--maxdepth directory-level-count] [(--ctime [+|-|=]posix-ctime)...] [(--atime [+|-|=]posix-atime)...] [(--mtime [+|-|=]posix-mtime)...] [--minsize minimum-byte-count] [--maxsize maximum-byte-count] [--size exact-byte-count] [(--exclude exclude-string)...] [(--prune prune-string)...] [--sjis] [(-o|--outfile) output-file-spec] [(-e|--logfile) log-file-spec] [-q|--quiet] [-v|--verbose] input-directory-path ...)
45
Description
Searches only for files with a POSIX ctime value that is after (+), before (-), or equal to (-) the specified time. If you omit this parameter altogether, arcfind searches for files or directories regardless of their ctime values. For information on specifying the parameter, including using multiple --ctime parameters to specify ranges, see Specifying dates and times on page 4-9.
--dirs
Returns a list of directories instead of a list of files. If you omit this parameter, arcfind returns a list of files. Tip: The --dirs parameter is useful when arcfind is used in conjunction with arcmkdir.
(-e|--logfile) log-file-spec
Writes diagnostics and statistics to the specified file. log-file-spec identifies a text file. If the file doesnt exist, arcfind creates it. If the file already exists, arcfind overwrites it without notifying you. In either case, the directory in which the file is located must already exist. For information on specifying the log file, see File specifications and directory paths on page 2-5. If you omit this parameter, arcfind sends log information to stderr. When youre using arcfind interactively, stderr is the window youre working in.
(--exclude exclude-string)... Searches only for files with names that do not include the
specified case-sensitive text string. For information on specifying text strings, see Text strings on page 4-11. If you omit this parameter, arcfind doesnt exclude files based on their names. Note: This parameter does not apply to directory names.
-h|--help -L|--link
Displays and describes the syntax for the arcfind command; then exits. Tells arcfind to follow symbolic links when searching a directory tree. If you omit this parameter, arcfind doesnt follow directory paths past any symbolic links it encounters. Note: This parameter is meaningful only on Unix systems. On Windows systems, arcfind always follows symbolic links.
46
Description
Searches only the specified number of levels down the directory trees, where the top directory in each is level one. For example, if the directory tree is A>B>C>D and --maxdepth is 3, arcfind returns a list of the files in A, B, and C or, if you include the --dirs parameter, arcfind returns a list containing the directories A, B, and C. Valid values for directory-level-count are integers greater than or equal to 1 (one). If you omit this parameter, arcfind searches each entire directory tree.
--maxsize maximum-byte-count
Searches only for files with a size less than or equal to the specified number of bytes. Valid values for maximum-bytecount are integers greater than or equal to 0 (zero). If you omit this parameter, arcfind searches for files without checking for a maximum size.
--minsize minimum-byte-count
Searches only for files with a size greater than or equal to the specified number of bytes. Valid values for minimum-bytecount are integers greater than or equal to 0 (zero). If you omit this parameter, arcfind searches for files without checking for a minimum size.
(--mtime [+|-|=]posix-mtime)...
Searches only for files with a POSIX mtime value that is after (+), before (-), or equal to (-) the specified time. If you omit this parameter altogether, arcfind searches for files or directories regardless of their mtime values. For information on specifying the parameter, including using multiple --mtime parameters to specify ranges, see Specifying dates and times on page 4-9. Note: HCP doesnt change the value of the mtime attribute. However, users and applications can change this value.
47
Description
Writes the list of selected files or directories to the specified file. output-file-spec identifies a text file. If the file doesnt exist, arcfind creates it. If the file already exists, arcfind overwrites it without notifying you. In either case, the directory in which the file is located must already exist. For information on specifying the output file, see File specifications and directory paths on page 2-5. If you omit this parameter, arcfind sends the list to stdout. When youre using arcfind interactively, stdout is the window youre working in. Tip: When you send the arcfind output to stdout, you can pipe it directly into the arcmv tool.
(--prune prune-string)...
Searches only directories with names that do not include the specified case-sensitive text string. For information on specifying text strings, see Text strings on page 4-11. When a directory is excluded, arcfind doesnt search its subdirectories either. If you omit this parameter, arcfind doesnt exclude directories based on their names.
-q|--quiet
Directs arcfind not to log processing and diagnostic information. This parameter is ignored if you specify the -v (--verbose) parameter. Searches only for files with a size exactly equal to the specified number of bytes. Valid values for exact-byte-count are integers greater than or equal to 0 (zero). If you omit this parameter, arcfind doesnt exclude files based on an exact size.
--size exact-byte-count
--sjis
Tells arcfind to check each input directory path for Shift JIS encoding. If arcfind encounters a path thats encoded that way, it converts the path to UTF-8 and continues processing. Logs detailed processing information in addition to the diagnostics and statistics generated by default. Displays the version number of the currently installed arcfind tool; then exits.
-v|--verbose --version
48
Description
Specifies the directory trees arcfind should search in a local or mounted file system. The path cannot be an HTTP or WebDAV URL. Use spaces to separate multiple directory paths. If a path includes spaces, enclose it in double quotation marks ("). The parameters used in an arcfind command apply to all the specified input directories. For information on specifying the input directory paths, see File specifications and directory paths on page 2-5 and arcfind output as arcmv input on page 4-11.
A datetime value starting with + matches times after the specified value. A datetime value starting with - matches times before the specified value. A datetime value starting with = matches only the specified date and time. This behavior is the default.
For details on specifying the date and time, see Datetime format on page 4-10. You can combine parameters to set time ranges. For details, see Using multiple time specifications on page 4-10.
49
Datetime format
The datetime format for the --atime, --ctime, and --mtime parameters is YYYY-MM-DD[ hh:mm:ss], where:
YYYY is the four-digit year. MM is the two-digit month. DD is the two-digit day. hh is the hour on a twenty-four hour clock. mm is the number of minutes past the hour. ss is the number of seconds past the minutes.
The year, month, and day, are separated by hyphens (-); the hour, minutes, and seconds, by colons (:). If you omit the time, arcfind uses 00:00:00 (midnight). The date and time must be separated by a space. When including the time, you need to either put the entire time specification in double quotation marks (""), including the leading character, if any (for example, "+2010-10-09 07:30:00"), or use a platform-specific escape character in front of the space (for example, +2010-10-09\ 07:30:00).
If you specify two parameters of the same type (for example, atime),
one with the plus sign (+) and one with the minus sign (-), arcfind returns the names of files with time values between the specified times.
If you specify two or more parameters of the same type (for example,
atime) with values starting with +, arcfind returns the names of files
410
If you specify two or more parameters of the same type (for example,
atime) with values starting with -, arcfind returns the names of files with time values before the earliest parameter value.
Also, if you specify an --mtime or --atime value along with a --ctime value, the specified --ctime value must be earlier than the --mtime or --atime value. If the --ctime value is later than either of these, arcfind returns an empty list.
Text strings
The text string in an --exclude or --prune parameter can be any combination of alphanumeric and special characters allowed by the platform-specific rules for files and directory names. If the string includes one or more spaces, you need to either put the entire string in double quotation marks (for example, "test C") or use a platform-specific escape character in front of each space (for example, test\ C). Text strings in Unix are case sensitive; those in Windows are not.
specified in the command line. If this is an absolute path, each path in the output is absolute as well. When you use the arcfind output as input to an arcmv command, arcmv generates the source and destination paths for the move operation by appending each path in the arcfind output to the source and destination locations specified in the arcmv command. The following conditions apply to the combination of paths:
If the output of the arcfind command consists of absolute paths and the
source location specified in the arcmv command is an absolute path, the resulting paths are invalid.
If you use two periods (..) to represent parent directories in the input
directory path for arcfind, each line in the output file begins with the double period. When arcmv concatenates these paths with a source location, the result is an invalid path. The paths specified in the two commands must combine correctly. For example:
411
The arcfind output must consist of paths that are relative to any source
directory specified by --src and --srcroot parameters in the arcmv command. Before running the arcfind command, you may need to set the working directory to ensure that the arcfind command generates output that the arcmv command can use.
When running arcfind in Windows, if you include the drive letter in the
input directory path, the drive letter is included in each file path in the output file. When you use this output as input to arcmv, the resulting file paths have the drive letter embedded in them (with no preceding delimiter). To prevent this problem, run arcfind on the drive on which the files you want are located so you dont need to include the drive letter in the input directory path. For more information about using the two commands together, see Using arcfind with arcmv on page 5-21. For examples that use arcfind output directly as input to an arcmv command, see Workflow examples on page 3-7.
by the command parameters. As a result, some combinations of parameters can produce unexpected results:
If you include the --size parameter with the --minsize and/or --maxsize
parameters, arcfind returns an error.
412
arcfind examples
arcfind examples
This section presents examples of the arcfind command. For examples that show the use of arcfind in conjunction with arcmv, see Workflow examples on page 3-7. Example 1 Heres the sample command:
Windows: arcfind --ctime "+2010-8-30 11:59:59" --ctime -2010-10-01 --outfile file-list.txt --logfile find-log.log --exclude .tmp pdfs Unix: arcfind --ctime "+2010-8-30 11:59:59" --ctime -2010-10-01 --outfile file_list --logfile find_log --exclude _tmp pdfs
Search for files starting in the pdfs directory, which is in the current
working directory
The metadata for the file was last modified during the month of September, 2010. The file name doesnt include the string .tmp (Windows) or _tmp (Unix).
Write the output list to a file named file-list.txt (Windows) or file_list (Unix)
in the current working directory
413
arcfind examples
Follow symbolic links (Windows does this automatically) End each directory search four levels down from the Employees or
Consultants directory
Not write any log information Write the output list to stdout (by default)
Example 3 Heres the sample command:
Windows: arcfind --minsize 50000 -o LargeFiles.txt -ve LargeFiles.log Employees\Claims Unix: arcfind --minsize 50000 -o large_files_out -ve large_files_log employees/claims
Search only for files that are larger than 50,000 bytes
414
arcfind examples
Search only for files that were stored in 2010 Write the output list to a file named Benefits-2010.txt (Windows) or
benefits_2010 (Unix) in the HCP Files (Windows) or hcp_files (Unix) subdirectory of the current working directory
415
arcfind examples
416
5
Moving files (arcmv)
The arcmv tool lets you copy or move files. The file source or destination can be a local or remote file system or a namespace, including the default namespace. In the arcmv command line, you tell arcmv which files to copy and where to put them. This chapter presents instructions for using arcmv.
51
About arcmv
About arcmv
The arcmv tool copies files from one location to another. By spawning multiple threads, arcmv copies or moves large numbers of files more efficiently than other copy or move tools. The source location for arcmv (that is, where the files are copied from) can be a local or remote file system or a namespace. The target location (that is, where the files are copied to) can be a local or remote file system or a namespace. However, if you copy or move files between two HCP namespaces, both namespaces must support identical login credentials. The default action for arcmv is copy. However, you can include the --rm parameter in the arcmv command line to remove the source files after copying them, effectively changing the action to move.
arcmv tries to write each source file to its target location. If a file with the
same name already exists at that location, the write fails. The exception to this is when the target location is in an HCP namespace with versioning enabled. In this case, the write creates a new version of the file. If the arcmv command line includes the --overwrite parameter, arcmv tries to delete the existing file before writing the new one. If the delete succeeds, the write replaces the deleted file with the source file. The exception to this is when the target location is in an HCP namespace with versioning enabled. In this case, if the delete succeeds, the write creates a new version of the file. Note: A namespace can be configured to prohibit reads, writes and/or deletes. If reads or writes are prohibited, HCP returns an error for each file arcmv tries read from or write to write to the namespace. If deletes are prohibited, arcmv cannot overwrite existing files. When the target location is a namespace and the namespace access protocol is HTTP, you can use the --md parameter to provide certain system metadata values for the files being copied. arcmv applies these values as it copies the files. When the target location is a namespace and the namespace access protocol is HTTP or WebDAV, you can provide a file containing custom metadata along with each source file. If the custom metadata is well-formed XML, arcmv stores it in the namespace along with the source file.
52
About arcmv
You can include parameters in the arcmv command line that limit the number of times arcmv can retry a copy or move operation, as well as the number of errors that can occur or the number of seconds that can elapse before arcmv terminates. You can include multiple source and target locations in an arcmv command. For information on how arcmv handles multiple locations, see Using single and multiple sources and targets on page 5-19.
arcmv input
Input to arcmv can be either a text file containing a list of any number of files or, alternatively, file names you enter interactively in a Windows command-prompt window or a Unix shell. In the arcmv command, you specify the location of the files to be copied by providing one or more URLs or absolute directory paths and, optionally, a relative path. For information on how arcmv constructs the source and target locations from your specifications, see Source and target file locations on page 5-17. To store custom metadata for a file, you include the name of the file containing the custom metadata in the same line as the source file name. By default, you use a comma to separate the two file names. However, you can override this default. You would do this, for example, if your file names contain commas. The custom metadata option is available only when the source files are listed in an input file, not when you supply them interactively. In the arcmv command, you specify the source location for custom metadata files separately from the source location for the files to be copied. This location must be a directory in a local or remote file system. You use the --cmdir parameter to specify this directory. If the --cmdir parameter is not present, arcmv assumes that each entire line in the input file identifies a single source file.
53
About arcmv
Heres an example of an input file that includes custom metadata for three of four source files:
Ramirez_John_2010-10--24, Ramirez\Projects.xml Wong_Lee_2010-9-18 ExampleConsultants\Smith_Mary_2010-10-11, msmith-projects ExampleConsultants\Gold_Susan_2010-10-11, sgold-projects
Note: A namespace can be configured to disallow the addition of custom metadata for files under retention. If the target namespace is configured this way and a source file would end up being under retention (either through the --md parameter or through inheritance), arcmv doesnt store the custom metadata for that file. Important: If you plan to pipe the output from the arcfind tool directly into arcmv, be sure to read arcfind output on page 4-3, Using arcfind with arcmv on page 5-21, and Source and target file locations on page 5-17 before you construct each command.
arcmv output
Output from arcmv is optional and consists of a list of the source files that were successfully copied. If arcmv successfully stored custom metadata for a source file, the output also lists the custom metadata file on the same line. The delimiter between the two file names is the same as the one in the input file. The output list shows each source file with its full path. For example, if the source location is C:\Corporate\Employees, the base directory is 2193_John_Doe, and the file name is status_2010_11_23, the listing in the output file would be:
C:\Corporate\Employees\2193_John_Doe\status_2010_11_23
If arcmv also copies or moves custom metadata for the file where, the --cmdir parameter specifies cust-metadata and the custom metadata file name in the input file is JohnDoe\Projects.xml, the listing in the output file would be:
C:\Corporate\Employees\2193_John_Doe\status_2010_11_23, cust-metadata\ JohnDoe\Projects.xml
54
About arcmv
arcmv log
In addition to the output list, arcmv generates a log containing statistics, diagnostics, and other processing information. You can specify whether the log is written to a file or displayed on-screen, as well as which types of information it contains. Tip: In Windows, use WordPad rather than Notepad to open arcmv log files. In WordPad, each listed file appears in a separate line. In Notepad, the files are listed in a single line. The statistics in the arcmv log are labeled and formatted for easy search and script postprocessing. The labels are:
File_Stats arcmv writes one File_Stats line to the log for each file it
processes. Each line has this format:
datetime message-level File_Stats: file-order outcome file-name size file-size time number-of-seconds secs
For example:
Oct 06 20:01:48.405 INFO File_Stats: 1 Success "log117.txt" size 987 time 0.010 secs
Note: arcmv doesnt write File_Stats lines for the files it skips.
55
About arcmv
datetime message-level Session_Stat: Time: hh:mm:ss Files: number-of-files-processed Good: number-of-successes Skipped: number-of-files-skipped Errors: number-of-errors Retries: number-of-retries Rates: total-number-of-bytes-copied KiB number-of-bytes-copied-per-second KiB/sec number-of-files-copied-per-second files/sec
For example:
Oct 06 20:01:49.407 INFO Session_Stat: Time: 00:00:01 Files: 2 Good: 2 Skipped: 0 Errors: 0 Retries: 0 Rates: 3.624 KiB 3.624 KiB/sec 2.00 files/sec
number-of-errors
number-of-retries The number of retries that have occurred since the session total-number-ofbytes-copied
The total number of bytes in the files arcmv has successfully copied since the session began, expressed in KiB (1,024 bytes).
56
About arcmv
(Continued)
Item
Description
number-of-bytes- The average number of bytes arcmv has successfully copied per second since the session began, expressed in KiB (1,024 copied-perbytes). second number-of-files-c The average number of files arcmv has successfully copied per opied-per-second second since the session began.
If the client has access to a directory containing individual certificate files, arcmv can check those files for a certificate representing the authority that signed the HCP certificate. The certificates in the directory must be in PEM format. The directory itself must have been processed using the c_rehash utility supplied with OpenSSL. You use the --https-capath parameter in the arcmv command to specify the directory containing the certificate files. This option is not available for Windows clients.
If the client has access to a file containing a list of certificates in PEM format, arcmv can check that list for a certificate representing the authority that signed the HCP certificate. This file can be the default certificate file for the client, or it can be a file that you specify. You use the --https-cacert parameter in the arcmv command to specify the file that lists the certificates.
57
About arcmv
In either case, if arcmv finds such a certificate, it proceeds with the requested operation. If it cannot find such a certificate, the requested operation fails.
Credentials HCP uses to verify that arcmv has permission to access the
namespace. These credentials consist of a username and password.
The types of operations that a client using the account can perform in
the namespace (permissions).
If these dont include the read permission, arcmv cannot read files from the namespace. If these dont include the write permission, arcmv cannot write files to the namespace. If they dont include the delete permission, arcmv cannot overwrite files in the namespace.
To get a data access account, see your namespace administrator. To use arcmv with an HCP namespace, you specify the data access account username and password in command parameters. You must also identify the hostname in the paths or use a --hostname parameter. For more information on these parameters, see arcmv syntax below
58
arcmv syntax
arcmv syntax
Heres the syntax for the arcmv command:
arcmv (-h|--help) |--version |([(--src source-location)...] [--srcroot source-path] (--dst destination-location)... [--dstroot destination-path] [--username user-name --password password [--hostname hostname]] [(--https-capath certificate-directory-path)| (--https-cacert certificate-file-spec)| --https-insecure] [--no-url-encode] [--md metadata-values] [--preserve-times] [--cmdir custom-metadata-path [--cmstrict]] [--delimiter "delimiter-character"] [--sjis] [--validate] [(-i|--infile) input-file-spec] [(-o|--outfile) output-file-spec] [--overwrite] [--rm] [(-r|--retry-limit) retry-count] [--error-limit error-count] [--time-limit number-of-seconds] [(-t|--threads) thread-count] [(-e|--logfile) log-file-spec] [(-l|--loglevel) log-level-number] [-q|--quiet] [-v|--verbose])
59
--cmstrict
Tells arcmv that each source file must be accompanied by a custom metadata file. If a source file has no accompanying custom metadata file, arcmv doesnt copy that source file. Also, if arcmv cannot successfully store the custom metadata for a source file, it doesnt copy that source file. This parameter is valid only if the arcmv command includes the --cmdir parameter. If you include this parameter without the --cmdir parameter, the arcmv command fails. Note: If the --cmstrict parameter is not present and arcmv cannot store the custom metadata for a source file, the source file is copied anyway.
--delimiter "delimiter-character"
Specifies the delimiter used between source file names and custom metadata file names in the input file. delimiter-character must be a single character enclosed in double quotation marks. Valid values are printable ASCII characters in the range 0x20 through 0x7E. If you omit this parameter, the delimiter is the comma (,).
510
Description
Specifies the target location for the files being copied. destination-location can be: A URL that identifies a directory in a namespace. The URL can use either a hostname or an IP address. If you use an IP address for an HCP namespace, you must also specify a --hostname parameter. An absolute or relative path to a directory in a local or remote file system.
This parameter is required. An arcmv command must include at least one --dst parameter and can have multiple. The arcmv command treats the locations specified by multiple --dst parameters as equivalents. For information on using this parameter, see Source and target file locations on page 5-17.
--dstroot destination-path
Adds the named relative path to the beginning of the specification of each file to be copied to the target location. For information on using this parameter, see Source and target file locations on page 5-17. Writes diagnostics and statistics to the specified file. log-file-spec identifies a text file. If the file doesnt exist, arcmv creates it. If the file already exists, arcmv overwrites it without notifying you. In either case, the directory in which the file is located must already exist. For information on specifying the log file, see File specifications and directory paths on page 2-5. If you omit this parameter, arcmv sends log information to stderr. When youre using arcmv interactively, stderr is the window youre working in.
--error-limit error-count
Tells arcmv not to start any new operations after the specified number of errors occur and to terminate after any current operations complete. As a result, some files may not be copied at all. Valid values for error-count are integers greater than or equal to 0 (zero). If you omit this parameter, arcmv doesnt terminate based on the number of errors that have occurred. For information on arcmv errors, see arcmv errors on page 5-26.
511
Description
Displays and describes the syntax for the arcmv command; then exits. The hostname for the HCP namespace to or from which you are moving or copying files, in this format:
namespace-name.tenant-name.hcp-name.domain-name
This parameter is required if you use an IP address in a source or destination URL for access to an HCP namespace. You do not use this parameter if the URL contains the hostname. This parameter is ignored for the default namespace.
--https-cacert certificate-file-spec
With HTTPS, tells arcmv to look in the specified file for an SSL certificate representing the authority that signed the certificate presented by HCP. For information on specifying the certificate file, see File specifications and directory paths on page 2-5. For more information on using this parameter, see arcmv with SSL security on page 5-7.
--https-capath certificate-directory-path
With HTTPS, tells arcmv to look in the specified directory for an SSL certificate representing the authority that signed the certificate presented by HCP. certificate-directory-path can be an absolute path or the path relative to the current working directory. To specify the current working directory itself, use a period (.) alone. This parameter is not valid in Windows. For more information on using this parameter, see arcmv with SSL security on page 5-7.
--https-insecure
With HTTPS, tells arcmv to accept the SSL certificate presented by HCP without checking whether its signed by a trusted authority. For more information on using this parameter, see arcmv with SSL security on page 5-7.
(-i|--infile) input-file-spec
Identifies the text file containing the list of files to be copied. For information on specifying the input file, see File specifications and directory paths on page 2-5. If you omit this parameter, arcmv waits for input from stdin. When youre using arcmv interactively, stdin is the window youre working in. To specify stdin explicitly in the arcmv command, use a hyphen (-) alone. For information on how arcmv concatenates the specifications of the files to be copied with the source and destination paths you specify in the arcmv command, see Constructing source and target paths on page 5-18.
512
Description
Tells arcmv how much processing information to log. Valid values for log-level are: 100 or FINEST Logs informational, warning, and error messages, along with detailed processing information 200 or FINER Logs informational, warning, and error messages, along with most of the detailed processing information 400 or FINE Logs informational, warning, and error messages, along with some of the detailed processing information 800 or INFO Logs informational, warning, and error messages (the default level) 900 or WARN Logs warning and error messages 1000 or SEVERE Logs only error messages
--md metadata-values
Stores the specified metadata values for each file copied to the target namespace. metadata-values has the form of an HTTP query containing one or more assignments of values to metadata properties, each separated from the next by an ampersand (&) (for example, shred=1&index=0). For more information on the --md parameter, see Metadata properties on page 5-22. HTTP rejects invalid query strings and returns a code indicating that an error has occurred. Tip: To avoid the possibility of the client operating system interpreting ampersands as control characters, precede each ampersand with the platform-specific escape character (for example, shred=1\&index=0). Note: This parameter is meaningful only when the target location is specified as a URL and arcmv is using the HTTP protocol. WebDAV does not support this feature.
513
Parameter --no-url-encode
Description
Tells arcmv not to URL-encode characters in HTTP and WebDAV source and target URLs. In each case, this applies to the entire URL, including elements specified in any --src, --srcroot, --dst, and --dstroot parameters, as well as in the input to arcmv. If you omit this parameter, arcmv URL-encodes all characters in source and target URLs. For more information on using the --no-url-encode parameter, see URL encoding on page 5-22.
(-o|--outfile) output-file-spec
Writes the list of copied files to the specified file. output-filespec identifies a text file. If the file doesnt exist, arcmv creates it. If the file already exists, arcmv overwrites it without notifying you. In either case, the directory in which the file is located must already exist. For information on specifying the output file, see File specifications and directory paths on page 2-5. If you specify a hyphen (-) alone as the value for this parameter, arcmv writes the list of copied files to stdout. When youre using arcmv interactively, stdout is the window youre working in. If you omit this parameter, arcmv doesnt list the copied files.
--overwrite
If arcmv tries to copy a file over an existing file and this parameter exists, it tries to delete the existing file and write the new copy. If arcmv is unable to delete an existing file (for example, if the retention period on the existing file doesnt allow it), it doesnt copy the file from the source location. If you omit this parameter, arcmv doesnt overwrite existing files. If a file with the same name exists, and the destination is not an HCP namespace with versioning enabled, arcmv doesnt copy the source file. Tip: To generate a list of the files that have been successfully copied, use the --outfile parameter.
--password
The password for the data access account specified by the --username parameter. This parameter is required to access an HCP namespace. This parameter is ignored for the default namespace.
514
Parameter --preserve-times
Description
Preserves the POSIX atime and mtime values of each file copied to the target namespace. If you omit this parameter, the atime and mtime values are both set to the time the file is copied to the namespace. Note: This parameter is meaningful only when the target location is specified as a URL and arcmv is using the HTTP protocol. WebDAV does not support this feature.
Suppresses the interactive display of the session log. Tells arcmv how many times to try again to copy a file if the action fails due to a nonfatal error. Valid values for retry-count are integers greater than or equal to 0 (zero). If you omit this parameter, arcmv doesnt try again to copy files after failures. Note: arcmv does not necessarily use the same --src and --dst parameters for each subsequent attempt.
--rm
Deletes each file from the source location after its successfully copied, in effect, changing the copy action to move. If you omit this parameter, arcmv doesnt delete any source files. Note: The --rm parameter applies only to source files and not to custom metadata files.
--sjis
Tells arcmv to check each source file specification in the input file for Shift JIS encoding. If arcmv encounters a file specification thats encoded using Shift JIS, it converts the specification to UTF-8 and continues processing. arcmv does not check custom metadata file specifications for Shift JIS encoding.
515
Description
Specifies the location of the files to be copied. source-location can be: A URL that identifies a directory in a namespace. The URL can use either a hostname or an IP address. If you use an IP address for an HCP namespace, you must also specify a --hostname parameter. An absolute or relative path to a directory in a local or remote file system.
An arcmv command can have multiple --src parameters; arcmv treats the locations they specify as equivalents. When the arcmv source is an HTTP location, the --src parameter is required. For information on using this parameter, see Source and target file locations on page 5-17.
--srcroot source-path
Adds the named path to the beginning of each file specification in the input list. For information on using this parameter, see Source and target file locations on page 5-17. Limits the number of concurrent arcmv threads to the specified number. Valid values for thread-count are integers in the range 1 (one) through the system-specific maximum threads per process. If you omit this value, arcmv starts ten threads that run concurrently.
-t|--threads thread-count
--time-limit number-of-seconds
Tells arcmv not to start any new operations after the specified number of seconds have passed and to terminate after any current operations complete. As a result, some files may not be copied at all. Valid values for number-of-seconds are integers greater than or equal to 0 (zero). If you omit this parameter, arcmv doesnt terminate based on a time limit.
--username
The username for the data access account youre using to copy or move files to or from an HCP namespace. This parameter is required to access an HCP namespace. This parameter is ignored for the default namespace.
-v|--verbose
Writes statistics for each file processed to the log file, if specified, or stderr, if not.
516
Parameter --validate
Description
Tells arcmv to validate each source file and custom metadata file it writes to the namespace. arcmv does this by generating a cryptographic hash value for the file and comparing this value to the cryptographic hash value HCP returns for the file. Note: This parameter is valid only when the target location is specified as a URL and arcmv is using the HTTP protocol.
--version
Displays the version number of the currently installed arcmv tool; then exits.
Each arcmv command must include at least one --dst parameter. The --src parameter is optional, but may be required to generate valid
input paths. Without it, arcmv treats relative paths in the input list as starting in the current working directory, which may not be correct. Also, using the --src parameter allows the arcmv output file to show the full directory path for each copied file.
517
relative to the source or destination location. Note: If the source and destination in an arcmv command have different paths but identify the same file (for example, if the paths differ only by IP address), using the --overwrite option results in the file being deleted. This happens because, to implement the --overwrite option, arcmv first deletes the target file and then copies the source file. In this case, because the source and target files are the same, the source no longer exists after arcmv deletes the target.
When constructing a target location, arcmv concatenates the --dst value, the --dstroot value (if any) and the file specification in the input file:
dst + \ + [ dstroot + \ +] file-specification
If the resulting paths are not absolute, they are relative to the current working directory. If any directories in the destination file paths dont exist, arcmv automatically creates them. When constructing paths, arcmv uses the elements exactly as you specify them. If a relative path includes the two periods that indicate the parent directory, arcmv puts those two periods in the file path, which results in an invalid path; for example:
http://finance.europe.hcp.example.com/rest + ..\Corporate\Consultants\ CurrentContracts + Ramirez_John_2010-10-14
becomes:
http://finance.europe.hcp.example.com/rest/../Corporate/Consultants/ CurrentContracts/Ramirez_John_2010-10-14
518
Use the DNS name of the HCP system in a single --src or --dst
parameter. For example:
--dst http://www.hcp.example.com/rest
In this case, HCP selects the IP address for each copy operation, using a round-robin technique to select the address to use for each operation. HCP also ensures that each request goes to an available address. However, this technique does not allow the client to cache the DNS results.
In this case, arcmv uses the specified locations in a straight round-robin; that is, it uses the first IP address for the first copy operation, the second IP address for the second operation, and so forth, starting again with the first IP address after it has run through the list. This manual round-robin approach does not ensure that the request is made to an available server. If a server is unavailable, an error occurs when arcmv tries to access the unavailable server.
519
In the output file (assuming arcmv successfully copies all the files):
http://192.168.1.1/rest/Corporate/Consultants/CurrentContracts/ Ramirez_John_2010-03--24 http://192.168.1.2/rest/Corporate/Consultants/CurrentContracts/ Wong_Lee_2010-10-18 http://192.168.1.3/rest/Corporate/Consultants/CurrentContracts/ ExampleConsultants/Smith_Mary_2010-08-11 http://192.168.1.1/rest/Corporate/Consultants/CurrentContracts/ ExampleConsultants/Gold_Susan_2010-08-11
520
Host identification requirements To access an HCP namespace, the HTTP requests generated by the arcmv command must identify the namespace and tenant. Also, arcmv supports only a single --hostname parameter. Therefore:
If the HCP namespace paths start with the namespace and tenant
names, you do not need to specify a --hostname parameter.
If the HCP namespace paths start with an IP address, you must specify
a --hostname parameter.
If you are copying or moving files between two HCP namespaces, you
cannot specify IP addresses for both the source and destination.
If you use relative paths in the arcfind command, ensure that any
arcmv command --src or --srcroot parameters specify the arcfind command working directory as the source directory.
If you use absolute paths in the arcfind command, ensure that the
resulting destination paths are valid and what you intended.
521
For more information about arcfind output as arcmv input, see arcfind output as arcmv input on page 4-11. For an example that uses arcfind output directly as input to an arcmv command, see Workflow examples on page 3-7. For more information on how arcmv constructs directory paths, see Source and target file locations on page 5-17.
URL encoding
HTTP and WebDAV require certain characters, such as spaces and plus signs (+), to be percent encoded. By default, arcmv performs this encoding on all characters in HTTP and WebDAV URLs. If directory or file names in the arcmv input or --src, --srcroot, --dst, and --dstroot parameters are already encoded, performing this encoding again changes those URLs, thereby causing unexpected results. This is particularly likely to be an issue when directory and file names include non-Latin characters. To prevent arcmv from percent-encoding URLs, include the --no-url-encode parameter in the command.
Metadata properties
The metadata properties that you can specify depend on the namespace type. The default namespace uses different properties from HCP namespaces.
522
For information on the values you can specify for these metadata properties, see Using a Namespace.
file_permissions The permissions that apply to the file directory_ permissions retention shred index
The permissions that apply to any directories arcmv creates automatically Specifies how long the file must remain in the namespace, in the form of retention setting that specifies a retention class or value Specifies whether the file is shredded when its deleted Specifies whether the file should be included in the search index
For information on the values you can specify for these metadata properties, see Using the Default Namespace.
523
Plan your directory structures before storing files. Make sure all
namespace users are aware of these plans.
Try to balance the namespace directory tree width and depth. Avoid structures that result in a single directory getting a large amount
of traffic in a short time. For example, if you ingest files rapidly, consider structures that do not store files by date and time.
If you do store files by date and time, consider the number of files
ingested during a given period of time. For example, if you ingest several hundred files per second, you might use a directory structure / year/month/day/hour/minute/second/. If you are going to ingest just a few files per second it would be better to use a less fine- grained structure.
Do not create directory structures that are more than 20 levels deep. Instead, create flatter directory structures. Avoid placing a large number of files (greater than 100,000) in a single directory. Instead, create multiple directories and evenly distribute the files among them.
New directories
If the directory path for a file to be copied doesnt exist in the target location, arcmv creates it and stores the file in it. With the WebDAV, CIFS, or NFS protocol, if the copy or move operation fails after arcmv has already created the directory, the new directory remains empty. With the HTTP protocol, arcmv doesnt create the directory at the target location if it cannot copy the file.
524
Optimizing performance
If the target of an arcmv operation is a namespace, spreading the processing load across the HCP system can help optimize arcmv performance. Here are some ways to do that:
Batch the source files into multiple input files. Then run multiple arcmv
sessions concurrently, each with a different input file. An added benefit to this technique is that if a session fails, you can rerun it using just the source files for that session.
525
Retrying on failure
The --retry-limit parameter tells arcmv how many times to try again to copy a file after a nonfatal error. Nonfatal errors are often due to a momentary hardware glitch or brief interruption in network service. When arcmv retries the copy, it is likely to succeed. You should always include the --retry-limit parameter in the arcmv command. Specify a large number of retries (for example, 100) to provide the most likelihood of success.
arcmv errors
You can use the --error-limit parameter to tell arcmv to terminate after a certain number of errors occur. Events that increment the error counter include:
arcmv cannot find a source file. arcmv cannot copy a source file to the target location. The arcmv command includes the --cmstrict parameter, and any of the
following applies:
526
The XML has a large number of different elements and attributes. In this case, try restructuring the XML to have fewer different elements and attributes. For example, try concatenating multiple element values, such as the different parts of an address, to create a new value for a single element. If you cannot restructure the XML to prevent failures, ask your namespace administrator about reconfiguring the namespace to prevent HCP from validating custom metadata XML.
527
arcmv examples
arcmv examples
This section presents examples of the arcmv command. For examples that show the use of arcmv in conjunction with arcfind, see Workflow examples on page 3-7. Example 1 Heres the sample command:
Windows: arcmv --src C:\MyDocs\Work --dst https://www.hcp.example.com/rest/BusDocs/Mktg --dstroot Whitepapers --infile ..\MoveWork.txt --outfile ..\MoveSuccess.txt --https-insecure --overwrite --rm -r 100 --threads 10 --time-limit 60 --loglevel 1000 Unix: arcmv --src /root/my_docs/work --dst https://www.hcp.example.com/rest/BusDocs/Mktg --dstroot whitepapers --infile ../move_work --outfile ../move_success --https-insecure --overwrite --rm -r 100 --threads 10 --time-limit 60 --loglevel 1000
528
arcmv examples
Delete and replace any existing file at the target location that has the
same name as a file being copied, provided the delete operation is allowed for the existing file
Delete each file from the source location after successfully copying it Retry the copy operation up to 100 times for any file for which nonfatal
errors occur
Use ten concurrent threads to perform the copy operations Not start any new copy operations after 60 seconds and terminate after
any current operations complete
Use http://finance.europe.hcp.example.com/rest/CorporateHQ/FinancialReports as
the target location (this identifies a directory in an HCP namespace)
529
arcmv examples
Add the relative path 20101024 to the beginning of each file specification
in the input file before copying the file to the target location
Set the file shred metadata value to true Retry the copy operation up to 50 times for any file for which nonfatal
errors occur
Not start any new copy operations if more than 25 nonfatal errors occur
and terminate after any current operations complete
Include individual file statistics in the arcmv log Log both file and session statistics to CopyLogs-2010-10\
Financials-2010-10-24.log (Windows) or copy_logs_2010_10/financials_2010_10_ 24_log (Unix) in the current working directory
530
arcmv examples
Use WebDAV as the namespace access protocol Add the relative path DistributionDiv\BOMs\Init-Service (Windows) or
distribution_div/BOMs/init_service (Unix) to the beginning of each file
specification in the input file before copying the file to the target location
Not copy any files that are not accompanied by custom metadata Use Z:\DistributionDiv\BOMs\Service-October-2010.txt (Windows) or /dist_div_
mt/distribution_div/BOMs/service_october_2010 (Unix) as the input file
Delete the original files from the source location after they are
successfully copied
Retry the copy operation up to 80 times for any file for which nonfatal
errors occur
531
arcmv examples
532
6
Creating directories (arcmkdir)
The arcmkdir tool creates empty directories in a default namespace or a local or remote file system. In the arcmkdir command line, you specify one or more target locations and one or more relative paths that end with a directory that does not exist in the target locations. This chapter presents instructions for using arcmkdir. Note: You cannot use arcmkdir to create directories in HCP namespaces.
61
About arcmkdir
About arcmkdir
The arcmkdir tool creates empty directories. The target location can be a default namespace or a local or remote file system. An arcmkdir command can include multiple target locations. For information on how arcmkdir handles multiple target locations, see arcmkdir target location processing on page 6-4.
arcmkdir constructs target locations the same way arcmv does. For
information on this, see Source and target file locations on page 5-17. Input to arcmkdir consists of one or more directory paths. arcmkdir creates any directories in each path that dont already exist. You can use arcmkdir local file systems and with WebDAV, CIFS, and NFS. You cannot use arcmkdir with HTTP. When using HTTPS with WebDAV, arcmkdir has the same options as arcmv for responding to the SSL server certificate HCP presents. For information on this, see arcmv with SSL security on page 5-7.
arcmkdir syntax
Heres the syntax for the arcmkdir command:
arcmkdir (-h|--help) |--version |((--dst destination-location)... [--dstroot destination-path] [(--https-capath certificate-directory-path)| (--https-cacert certificate-file-spec)| --https-insecure] [--no-url-encode] [-v|--verbose] new-directory-path ...)
62
Specifies the target location for the directories being created. (--dst destination-location)... destination-location can be: A URL that identifies a directory in a default namespace. The URL can use either the DNS name of the HCP system or an IP address for the system. The URL must specify a WebDAV address. An absolute or relative path to a directory in a local or remote file system. Relative paths are rooted in the working directory.
This parameter is required. An arcmkdir command must include at least one --dst parameter and can have multiple. For information on using multiple parameters, see --dst arcmkdir target location processing on page 6-4.
--dstroot destination-path Adds the named relative path to each directory path to be created in
the target location, following the directory or directories specified in the --dst parameter(s).
Displays and describes the syntax for the arcmkdir command; then exits. With HTTPS, tells arcmkdir to look in the specified file for an SSL certificate representing the authority that signed the certificate presented by HCP. For information on specifying the certificate file, see File specifications and directory paths on page 2-5. For more information on using this parameter, see arcmv with SSL security on page 5-7.
--https-capath certificate-directorypath
With HTTPS, tells arcmkdir to look in the specified directory for an SSL certificate representing the authority that signed the certificate presented by HCP. certificate-directory-path can be an absolute path or the path relative to the current working directory. To specify the current working directory itself, use a period (.) alone. You cannot use this parameter with Windows. For more information on using this parameter, see arcmv with SSL security on page 5-7.
--https-insecure
With HTTPS, tells arcmkdir to accept the SSL certificate presented by HCP without checking whether its signed by a trusted authority. For more information on using this parameter, see arcmv with SSL security on page 5-7.
63
Parameter --no-url-encode
Description
Tells arcmkdir not to URL-encode characters in URLs. In each case, this applies to the entire URL, including elements specified in any --dst and --dstroot parameters, as well as in the new directory path. If you omit this parameter, arcmkdir URL-encodes all characters in URLs. For more information on using the --no-url-encode parameter, see URL encoding on page 5-22.
Writes informational, warning, and error messages and detailed processing information to stderr. Displays the version number of the currently installed arcmkdir tool; then exits. Specifies the directories arcmkdir should create. Use spaces to separate multiple directory paths.
Use the DNS name of the HCP system in a single --dst parameter. For
example:
--dst http://www.hcp.example.com/webdav/fcfs_data
64
arcmkdir examples
In this case, HCP selects the IP address for each create operation, using a round-robin technique. HCP ensures that each request goes to an available server. However, this technique does not allow the client to cache the DNS results.
Use multiple --dst parameters with different IP addresses for the HCP
system. For example:
--dst http://192.168.1.1/webdav/fcfs_data --dst http://192.168.1.2/webdav/fcfs_data --dst http://192.168.1.3/webdav/fcfs_data
In this case, arcmkdir uses the specified locations in a straight roundrobin; that is, it uses the first IP address for the first create operation, the second IP address for the second operation, and so forth, starting again with the first IP address after it has run through the list. This manual round-robin approach does not ensure that the request is made to an available address.
arcmkdir examples
The section presents examples of the arcmkdir command. Example 1 Heres the sample command:
Windows: arcmkdir --dst X:\BusDocs\Mktg --dstroot Whitepapers Generic\InHouse Generic\ThirdParty Unix: arcmkdir --dst /busdocs_mnt/busdocs/mktg --dstroot whitepapers generic/inhouse generic/third_party
This command tells arcmkdir to create two or three new directories in X:\BusDocs\Mktg\Whitepapers (Windows) or /busdocs_mnt/busdocs/mktg/ whitepapers (Unix):
65
arcmkdir examples
directories
66
7
HCP client tools use cases
Hitachi Data Systems customers represent a variety of business, industry, and research environments. These customers are using HCP client tools in conjunction with HCP to address their differing data storage needs. This chapter describes two real-life use cases in which HDS customers are successfully using HCP client tools to support their data storage strategies.
HCP client tools use cases Using the HCP Client Tools
71
The challenge
The challenge in this case was to develop software that would automatically store new image files on a periodic basis. The three propositions embodied in that challenge are:
The software must be able to run without human intervention. The software must be able to use time criteria in selecting files to be
stored.
The software could not require any changes to their existing software
and processes.
The software could not overwrite image files already existing in the
target location.
72
HCP client tools use cases Using the HCP Client Tools
Successful completion of the operation. Warnings for all files that were selected for storage but not copied because they already existed in the target location. The applicable researcher would review these files and decide, on an individual basis, whether the old or new image was correct. Errors that occurred.
Note: The organization used a third-party data migration tool for the initial transfer of the existing image files to HCP.
The solution
The solution in this case entailed:
The arcfind command to search this centralized storage area and select the files modified since the last time the script ran The arcmv command to copy the selected files to the default namespace in the HCP system.
HCP client tools use cases Using the HCP Client Tools
73
operation. Before each execution, the script looks up the timestamp of the last successful operation and uses it as the value of the --mtime parameter with the plus sign (+) prefix in the arcfind command it builds. This tells arcfind to search the input directories for all files created or modified since the last successful operation. Like the input directory paths for arcfind, the source and target locations for the arcmv command, as well as the email address to which the script should send the results of the copy operation, are passed to the script as arguments. The arcmv tool creates a log file so the Python script can parse the results and email success and failure information to the designated addresses. The name of this file is also encoded in the script. Because the arcfind and arcmv tools run one after the other with no intervening processing, the script sets up a command line in which the output from arcfind is piped directly into arcmv. Heres an example of what such a command line looks like:
..\..\ArcTools\arcfind --mtime "+2010-10-14 03:00:00" ImageBase\NorthAmerica ImageBase\SouthAmerica ImageBase\CentralAmerica ImageBase\Europe ImageBase\Asia ImageBase\Africa ImageBase\Oceania ImageBase\ArcticRegion ImageBase\Antarctica | ..\..\ArcTools\arcmv --src C:\Research\Catalog --dst http://www.hcp-hi-res.flora.org/fcfs_data/Catalog --loglevel 900 --verbose --logfile ..\ImageReplication.log
74
HCP client tools use cases Using the HCP Client Tools
The challenge
The challenge in this case was to:
The images of any given item could not be moved to the namespace
until the microscope had captured them all.
The solution
The solution to this problem was a Unix shell script that used each of the three HCP client tools. In this script, which runs at regularly scheduled intervals on each researchers computer:
The arcfind tool selects all the image files last modified within a
specified amount of time before the current time. This ensures that the resulting list of files doesnt include only a partial set of images for any given item.
The arcmv tool moves each file in the arcfind list to the default
namespace in the HCP system. Because arcmv moves the files instead of copying them, the local drive never completely fills up. In fact, it appears to have unlimited space.
The arcfind tool selects all the directories last modified within a
specified amount of time before the current time.
The arcmkdir tool creates a new directory in the namespace for each
directory in the arcfind list. Note: Because the solution entails the use of the arcmkdir tool to create directories in the namespace, the customer needed to use the default namespace.
HCP client tools use cases Using the HCP Client Tools
75
76
HCP client tools use cases Using the HCP Client Tools
Install the client tools executables Compile the client tools source code
Appendix1
Using the HCP Client Tools
Compiled-executable platforms
Compiled-executable platforms
The HCP client tools come as compiled executables for these platforms:
Sun Solaris 10 SPARC Red Hat Enterprise Linux ES 5 (32 bit) IBM AIX 5.3 HP-UX 11i v1 (11.11) on PA-RISC
Variable
Appendix-2
Installing the HCP client tools Using the HCP Client Tools
Platform
AIX HP-UX LIBPATH SHLIB_PATH
Variable
This creates a subdirectory named arctools-source in the current directory. 2. Change to the created subdirectory:
cd arctools-source
3. Enter this command to configure the build environment for the client tools:
./configure
Installing the HCP client tools Using the HCP Client Tools
Appendix-3
If any of the required third-party libraries are not installed on your system, the configuration request fails. The failure message lists the libraries that are missing. 4. Enter this command to build the client tools:
make install
This builds the client tools executables and stores them in the arctoolssource directory.
Appendix-4
Installing the HCP client tools Using the HCP Client Tools
Glossary
A
access protocol
See namespace access protocol.
arcfind
The HCP client tool used to generate a list of files that meet a set of specified criteria.
arcmkdir
The HCP client tool used to create empty directories in a local or remote file system or in the default namespace in an HCP system.
arcmv
The HCP client tool used to copy or move files from a location on a client computer or a namespace in an HCP system to another location on a client computer or another namespace.
atime
Metadata that initially specifies the date and time at which a file was last accessed. Users and applications can change this metadata, thereby causing it to no longer reflect the actual access time. Note: This is not the normal POSIX usage for atime.
Glossary1
Using the HCP Client Tools
CIFS
C
CIFS
Common Internet File System. One of the protocols HCP uses to provide access to the contents of the default namespace. CIFS lets Windows clients access files on a remote computer as if they were part of the local file system.
client tools
See HCP client tools.
ctime
POSIX metadata that specifies the date and time of the last change to the metadata for a file
custom metadata
One or more user-defined properties that provide descriptive information about a file in HCP. Custom metadata, which is normally specified as XML, enables future users and applications to understand and repurpose file content.
D
data access account
A set of credentials that give a user or application access to one or more HCP namespaces. For each namespace, the account specifies which operations the user or application can perform.
Data Migrator
See HCP Data Migrator (HCP-DM).
default namespace
A namespace that supports the HTTP, WebDAV, CIFS, NFS, SMTP, and NDMP protocols and does not require user authentication for data access. An HCP system can have at most one default namespace.
DNS
See domain name system (DNS).
domain
A group of computers and devices on a network that are administered as a unit.
Glossary2
Using the HCP Client Tools
HCP namespace
F
fixed-content data
A digital asset ingested into HCP and preserved in its original form. Once stored, fixed-content data cannot be modified.
G
GID
Group identifier.
H
HCP
See Hitachi Content Platform (HCP).
HCP-DM
See HCP Data Migrator (HCP-DM).
HCP namespace
A namespace that requires user authentication for data access. An HCP system can have multiple HCP namespaces.
Glossary3
Using the HCP Client Tools
HTTP
Hypertext Transfer Protocol. One of the protocols HCP uses to provide access to the contents of a namespace.
HTTPS
HTTP with SSL security. See HTTP and SSL.
I
index
See search index.
index setting
The property that specifies whether a file should be indexed.
M
metadata
System-generated and user-supplied information about a file.
mtime
Metadata that specifies the date and time of the last change to the file data. Users and applications can change this metadata, thereby causing it to no longer reflect the actual change time.
Glossary4
Using the HCP Client Tools
POSIX
N
namespace
A logical partition of the files stored in an HCP system. A namespace consists of a grouping of files such that the files in one namespace are not visible in any other namespace. Namespaces are configured independently of each other and, therefore, can have different properties.
NFS
Network File System. One of the protocols HCP uses to provide access to the contents of the default namespace. NFS lets clients access files on a remote computer as if they were part of the local file system.
P
permission
One of these:
In POSIX permissions, the ability granted to the owner, the members of a group, or other users to access a file, directory, or symbolic link. A POSIX permission can be read, write, or execute. In a data access account, the granted ability to perform a specific type of operation in a given namespace.
policy
One or more settings that influence how transactions and services work on files in HCP. Such a setting can be a property of a file, such as retention, or a property of a namespace, such as versioning.
POSIX
Portable Operating System Interface for UNIX. A set of standards that define an application programming interface (API) for software designed to run under heterogeneous operating systems.
Glossary5
Using the HCP Client Tools
protocol
protocol
See namespace access protocol.
R
repository
The aggregate of the namespaces defined for an HCP system.
retention class
A named retention setting.
retention period
The period of time during which a file stored in HCP cannot be deleted.
retention setting
The property that determines the retention period for a file.
S
search facility
An interface between the search functionality provided by a system such as HDDS or HCP and the HCP Search Console. Only one search facility can be enabled at any given time.
search index
An index of the metadata and key terms in files. The active search system builds, maintains, and stores this index.
service
A background process that performs a specific function that contributes to the continuous tuning of the HCP system. In particular, services are responsible for optimizing the use of system resources and maintaining the integrity and availability of the data stored in the HCP repository.
shred setting
The property that determines whether a file will be shredded or simply removed when its deleted from HCP.
Glossary6
Using the HCP Client Tools
WebDAV
shredding
The process of deleting a file and overwriting the locations where its bytes were stored in such a way that none of its data or metadata can be reconstructed. Also called secure deletion.
SSL
Secure Sockets Layer. A key-based Internet protocol for transmitting documents through an encrypted link.
system metadata
System-managed properties that describe the content of a file. System metadata includes policies, such as retention and data protection level, that influence how transactions and services affect the file.
T
tenant
An administrative entity created for the purpose of owning and managing namespaces and data access accounts. Tenants typically correspond to customers, business units, or individuals.
U
UID
User ID.
Unix
Any UNIX-like operating system (such as UNIX itself or Linux).
V
versioning
A feature that allows the creation and management of multiple versions of a file.
W
WebDAV
Web-based Distributed Authoring and Versioning. One of the protocols HCP uses to provide access to the contents of the default namespace. WebDAV is an extension of HTTP.
Glossary7
Using the HCP Client Tools
WORM
WORM
Write once, read many. A data storage property that protects the stored data from being modified or overwritten.
Glossary8
Using the HCP Client Tools
Index
Symbols
as keyword prefix 2-4 representing stdin 5-12 representing stdout 5-14 with time parameters 4-9 -- as keyword prefix 2-4 + with time parameters 4-9 .directory_metadata 6-4 = with time parameters 4-9 syntax 4-5 time-based file selection 4-94-11 in time-critical image storage use case 7-6 arcmkdir See also HCP client tools; command lines about 2-2, 6-2 advantages 3-4 command considerations 6-46-5 examples 6-56-6 with HTTPS 6-2, 6-3 input 6-2 namespace access protocols supported 6-2 parameter descriptions 6-36-4 reserved directory name 6-4 round-robin processing 6-46-5 syntax 6-2 target locations 6-2, 6-3 in time-critical image storage use case 7-6 URL encoding 5-22 arcmv See also HCP client tools; copying files; moving files about 2-2, 5-25-3 accessing HCP namespaces with 5-8, 5-20 5-21 advantages 3-3 with arcfind 4-114-12, 5-215-22 authenticated access 5-8 command considerations 5-175-26 custom metadata 5-2 errors 5-26 examples 5-285-31 host name considerations 5-21 with HTTPS 5-75-8, 5-12 in image-file replication use case 7-37-4 input 5-35-4, 5-12 log 5-55-7 metadata 5-2, 5-13 optimizing performance 5-25
A
absolute directory paths 2-5 access protocols See namespace access protocols advantages of HCP client tools 3-33-4 of scripts 3-63-7 alternative tools 3-23-4 arcfind See also HCP client tools; command lines; finding files about 2-2, 4-2 advantages 3-3 with arcmv 4-114-12, 5-215-22 command considerations 4-94-13 examples 4-134-15 HCP as file source 4-15 how it works 4-2 in image-file replication use case 7-37-4 input directories 4-9 logs 4-34-4 multiple time specifications 4-104-11 output 4-3 parameter descriptions 4-54-9 parameter interactions 4-124-13 piping output into arcmv 3-4, 4-2 Shift JIS encoding 4-8 size-based file selection 4-7, 4-8
Index1
Using the HCP Client Tools
arcmv, output
output 5-4 parameter descriptions 5-105-16 parameter interactions 5-26 preserving atime and mtime 5-15 protocol choice 5-22 round-robin processing 5-19 Shift JIS encoding 5-15 source locations 5-25-3, 5-16 source locations, constructing 5-175-20 syntax 5-9 target locations 5-25-3, 5-11 target locations, constructing 5-175-20 in time-critical image storage use case 7-6 URL encoding 5-22 user name and password considerations 5-20 validating files 5-17 arguments 2-4 atime parameter, arcfind about 4-5 datetime format 4-10 multiple 4-104-11 with other time parameters 4-11 specifying 4-94-11 atime, preserving with arcmv 5-15 authenticated access, arcmv 5-8 See also hostname parameter, arcmv; password parameter, arcmv; username parameter, arcmv Avg/file (in arcfind logs) 4-4 command considerations arcfind 4-94-13 arcmkdir 6-46-5 arcmv 5-175-26 command conventions for HCP client tools 2-3 2-6 command lines See also HCP client tools; arcfind; arcmkdir; arcmv; parameters arguments 2-4 case sensitivity 2-4 directory paths 2-52-6 file specifications 2-52-6 HCP client tools 2-3 keyword parameters 2-3 keyword prefixes 2-4 parameters 2-32-4 separators 2-4 structure 2-4 command-line tools See HCP client tools; command lines common parameters 2-5 compiling client tools source code A-3A-4 concatenating single-letter keyword parameters 2-4 constructing source locations 5-175-20 target locations 5-175-20, 6-2 conventions HCP client tool commands 2-32-6 HCP directory structure 3-5 copying files arcmv 5-15-31 tools for 3-33-4 creating directories with arcmv 5-24 directories, tools for 3-4 empty directories with arcmkdir 6-16-6 lists of directories 4-14-15 lists of files 4-14-15 lists of files, tools for 3-43-5 criteria for finding files 3-43-5 ctime parameter, arcfind about 4-6 datetime format 4-10 examples 4-13, 4-15 multiple 4-104-11 with other time parameters 4-11 specifying 4-94-11 custom metadata See also metadata; system metadata about 1-3 in input file 5-35-4 location 5-10
C
case sensitivity commands 2-4 text strings 4-11 choosing namespace access protocols for copy or move 3-6 tools for copy or move 3-6 tools for find 3-5 CIFS protocol 2-2, 5-22 client tools compiling source code A-3A-4 installing executables A-2A-3 platforms A-2 See HCP client tools cmdir parameter, arcmv See also custom metadata about 5-10 example 5-305-31 cmstrict parameter, arcmv See also custom metadata about 5-10 example 5-305-31
Index2
Using the HCP Client Tools
files
in output listing 5-4 requiring 5-10 storing for files 5-2 arcmv 5-11 arcmv examples 5-285-29, 5-295-30, 5-305-31 constructing target locations 5-175-18
D
data See files data access accounts 5-8 dates and times, specifying 4-94-11 datetime in arcmv file statistics 5-5 in arcmv session statistics 5-6 format 4-10 parameter interactions 4-11 deciding where to put files in a namespace 3-5 default namespace See also HCP namespaces; namespaces about 1-2 metadata properties 5-23 deletions prohibited 5-2 delimiter parameter, arcmv 5-10 See also custom metadata depth of directory trees, arcfind maximum 4-7 dir_permissions metadata property 5-23 directories as arcfind input 4-9 arcfind output as arcmv input 4-114-12, 5-215-22 creating empty with arcmkdir 6-16-6 creating with arcmv 5-24 generating lists of 4-14-13, 4-14, 7-6 pruning 4-8 reserved name 6-4 specifying 2-52-6 structure in HCP 3-5, 5-24 tools for creating 3-4 Directories processed (in arcfind logs) 4-3 dirs parameter, arcfind about 4-6 examples 4-14, 7-6 with maxdepth parameter 4-12 Distribution of File Sizes (in arcfind logs) 4-4 DNS manager 5-19 dst parameter arcmkdir 6-3 arcmkdir examples 6-56-6, 7-6 arcmv 5-11 arcmv examples 5-285-29, 5-295-30, 5-305-31, 7-4, 7-6 constructing target locations 5-175-18 multiple 5-11 dstroot parameter arcmkdir 6-3 arcmkdir example 6-5
E
e parameter See also logfile parameter; logs arcfind 4-6 arcfind examples 4-144-15, 7-6 arcmv 5-11 arcmv examples 5-295-30, 5-305-31, 7-6 empty directories creating with arcmkdir 6-16-6 creating with arcmv 5-24 encoding URLs 5-22 equal signs with time parameters 4-9 error-limit parameter, arcmv 5-26 about 5-11 example 5-295-30 errors arcmv 5-26 limiting in arcmv 5-11 Errors (in arcmv session statistics) 5-6 escape character in datetime format 4-10 in text strings 4-11 examples about 2-6 arcfind 4-134-15 arcmkdir 6-56-6 arcmv 5-285-31 scripts 3-73-9 use cases 7-17-6 workflow 3-73-9 exclude parameter, arcfind about 4-6 example 4-13 multiple 4-12 text strings 4-11
F
file statistics in arcmv logs 5-5 file_permissions metadata property 5-23 File_Stats (in arcmv file statistics) 5-5 file-name (in arcmv file statistics) 5-5 file-order (in arcmv file statistics) 5-5 files See also logs copied by arcmv 5-35-4 copying 5-15-31 deciding where to put 3-5 finding 4-14-15 identifying for copy or move 3-43-5
Index3
Using the HCP Client Tools
files, moving
moving 5-15-31 overwriting 5-2, 5-14 size-based selection 4-7, 4-8 specifying 2-52-6 tools for copying 3-33-4 tools for finding 3-3 tools for moving 3-33-4 Files (in arcmv session statistics) 5-6 Files processed (in arcfind logs) 4-4 files/sec (in arcmv session statistics) 5-7 file-size (in arcmv file statistics) 5-5 -file-spec in syntax 2-5 filtered (in arcfind logs) 4-4 final delimiter in directory paths 2-5 finding files See also arcfind arcfind 4-14-15 criteria for 3-43-5 in HCP 4-2 tools for 3-3 FINE log level 5-13 FINER log level 5-13 FINEST log level 5-13 fixed-content storage system 1-2 following symbolic links 4-6 HCP client tools See also arcfind; arcmkdir; arcmv; command lines about 2-2 advantages 3-33-4 alternatives 3-23-4 arcfind 4-14-15 arcmkdir 6-16-6 arcmv 5-15-31 command conventions 2-32-6 common parameters 2-5 compared with HCP-DM 3-2 directory paths 2-52-6 file specifications 2-52-6 relationship to HCP 2-3 use cases 7-17-6 using 3-13-9 HCP Data Migrator, compared with HCP client tools 3-2 HCP namespaces See also namespaces; default namespace about 1-2 accessing using arcmv 5-8, 5-205-21 metadata properties 5-23 HCP-DM, compared with HCP client tools 3-2 help parameter about 2-5 arcfind 4-6 arcmkdir 6-3 arcmv 5-12 hh:mm:ss (in arcmv session statistics) 5-6 Hitachi Content Platform See HCP hold metadata property 5-23 hostname parameter, arcmv 5-12, 5-21 HTTP protocol about 2-2 arcfind restriction 4-2 arcmkdir restriction 6-2 with arcmv 5-22 backslash handling 2-6 preserving times 5-15 specifying metadata 5-10, 5-13 validating files 5-17 HTTPS arcmkdir 6-2, 6-3 arcmv 5-75-8, 5-12 https-cacert parameter arcmkdir 6-3 arcmv 5-12 https-capath parameter arcmkdir 6-3 arcmv 5-12
G
generating See also arcfind lists of directories 4-14-13, 4-14, 7-6 lists of files 4-14-15 lists of files, tools for 3-43-5 GiB 4-4 gid metadata property 5-23 Good (in arcmv session statistics) 5-6
H
h parameter See also help parameter arcfind 4-6 arcmkdir 6-3 arcmv 5-12 HCP about 1-11-2 as arcfind source 4-15 finding files 4-2 IP address selection for arcmv 5-19 IP address selection for directory creation 6-46-5 relationship to HCP client tools 2-3 source location processing 5-19 target location processing 5-19, 6-46-5
Index4
Using the HCP Client Tools
I
i parameter, arcmv See also infile parameter, arcmv about 5-12 examples 5-295-30, 5-305-31 identifying files to move or copy 3-43-5 image-file replication use case 7-27-4 index metadata property 5-23 infile parameter, arcmv See also i parameter about 5-12 example 5-285-29 INFO log level 5-13 input arcmkdir 6-2 arcmv 5-35-4, 5-12 input file, arcmv custom metadata 5-35-4 delimiter 5-10 installing client tools executables A-2A-3 interactions arcfind parameters 4-124-13 arcmv parameters 5-26 introduction to HCP client tools 2-12-6 IP addresses arcmkdir round-robin processing 6-46-5 arcmv round-robin processing 5-19
K
keyword parameters about 2-3 prefixes 2-4 required 2-4 KiB 4-4 KiB/sec (in arcmv session statistics) 5-7
M
max depth (in arcfind logs) 4-3 max passing entries (in arcfind logs) 4-4 maxdepth parameter, arcfind about 4-7 with dirs parameter 4-12 example 4-14 with symbolic links 4-12 maxsize parameter, arcfind about 4-7 with other size parameters 4-12
L
L parameter, arcfind about 4-6 example 4-14
Index5
Using the HCP Client Tools
md parameter, arcmv
md parameter, arcmv See also metadata about 5-13 example 5-295-30 message-level in arcmv file statistics 5-5 in arcmv session statistics 5-6 metadata See also system metadata; custom metadata about 1-3 properties 5-225-23 specifying with arcmv 5-2, 5-13 MiB 4-4 minsize parameter, arcfind about 4-7 example 4-144-15 with other size parameters 4-12 minus signs as keyword prefixes 2-4, 2-4 representing stdin 5-12 representing stdout 5-14 with time parameters 4-9 moving files arcmv 5-15-31 rm parameter 5-2, 5-15 tools for 3-33-4 mtime parameter, arcfind about 4-7 datetime format 4-10 examples 7-4, 7-6 multiple 4-104-11 with other time parameters 4-11 specifying 4-94-11 in time-critical image storage use case 7-3 7-4, 7-6 mtime, preserving with arcmv 5-15 multiple dst parameters 5-11 exclude parameters 4-12 prune parameters 4-12 src parameters 5-16 time specifications in arcfind 4-104-11 deciding where to put files 3-5 deletions prohibited 5-2 directory structure 3-5, 5-24 writes prohibited 5-2 NFS protocol 2-2, 5-22 no-url-encode parameter arcmkdir 6-4 arcmv 5-14 URL encoding 5-22 number-of-bytes-copied-per-second (in arcmv session statistics) 5-7 number-of-errors (in arcmv session statistics) 5-6 number-of-files-copied-per-second (in arcmv session statistics) 5-7 number-of-files-processed (in arcmv session statistics) 5-6 number-of-files-skipped (in arcmv session statistics) 5-6 number-of-retries (in arcmv session statistics) 5-6 number-of-seconds (in arcmv file statistics) 5-5 number-of-successes (in arcmv session statistics) 5-6
O
o parameter See also outfile parameter arcfind 4-8 arcfind example 4-144-15 arcmv 5-14 arcmv examples 5-295-30, 5-305-31 optimizing arcmv performance 5-25 others (in arcfind logs) 4-4 outcome (in arcmv file statistics) 5-5 outfile parameter See also o parameter arcfind 4-8 arcfind example 4-13, 4-15 arcmv 5-14 arcmv example 5-285-29 output arcfind 4-3, 4-8 arcmv 5-4, 5-14 overwrite parameter, arcmv about 5-2, 5-14 example 5-285-29 overwriting files 5-2, 5-14
N
namespace access protocols See also HTTP protocol; WebDAV protocol choosing for copy or move 3-6 supported 2-2 supported by arcmkdir 6-2 namespaces See also HCP namespaces; default namespace about 1-2
P
parameters See also command lines about 2-32-4
Index6
Using the HCP Client Tools
R
r parameter, arcmv 5-15 See also retry-limit parameter, arcmv Rates (in arcmv session statistics) 5-6 relative directory paths 2-5 replication use case 7-27-4 required keyword parameters about 2-4 arcmkdir 6-3 arcmv 5-11 arcmv used with HCP namespaces 5-12, 5-14, 5-16 reserved names, .directory_metadata 6-4 retention metadata property 5-23 Retries (in arcmv session statistics) 5-6 retry-limit parameter, arcmv about 5-15 example 5-295-30 retrying on failure 5-26 rm parameter, arcmv about 5-2, 5-15 examples 5-285-29, 5-305-31, 7-6 in time-critical image storage use case 7-6 round-robin processing arcmkdir 6-46-5 arcmv 5-19
S
scripts advantages 3-63-7 examples 3-73-9 for image-file replication use case 7-37-4 for time-critical image storage use case 7-6 searching See finding secs (in arcmv file statistics) 5-5 separators in command lines 2-4 in datetime format 4-10 session statistics in arcmv logs 5-65-7 Session_Stat (in arcmv session statistics) 5-6 Session_Stats (in arcmv session statistics) 5-7 SEVERE log level 5-13 shell script for time-critical image storage use case 7-6 Shift JIS encoding arcfind 4-8 arcmv 5-15 shred metadata property 5-23
Q
q parameter See also logs; quiet parameter arcfind 4-8 arcfind examples 7-6 arcmv 5-15 arcmv example 7-6 quiet parameter See also logs; q parameter about 2-5 arcfind 4-8 arcfind example 4-14 arcmv 5-15 with loglevel parameter 5-26 with logfile parameter in arcfind 4-13
Index7
Using the HCP Client Tools
T
t parameter, arcmv 5-16 See also threads parameter, arcmv target locations arcmkdir 6-2 arcmkdir processing 6-46-5 arcmv 5-25-3 arcmv processing 5-19 constructing 5-175-20, 6-2 specifying 5-11, 6-3 tenants 1-2 text strings 4-11 threads parameter, arcmv about 5-16 examples 5-285-29, 7-6 optimizing performance 5-25 time in arcmv file statistics 5-5 in datetime parameters 4-10 limiting in arcmv 5-16 Time (in arcmv session statistics) 5-6 time specifications, multiple 4-104-11 time-critical image storage use case 7-47-6 time-limit parameter, arcmv about 5-16 example 5-285-29 times, specifying 4-94-11 tools See also HCP client tools alternative 3-23-4 choosing for copy or move 3-5, 3-6 choosing for find 3-5 for copying files 3-33-4 for creating directories 3-4 for finding files 3-3 for moving files 3-33-4 Total passed size (in arcfind logs) 4-4 total-number-of-bytes-copied (in arcmv session statistics) 5-6
U
uid metadata property 5-23 Unix arcfind examples 4-134-15 arcmkdir examples 6-56-6 arcmv examples 5-285-31
Index8
Using the HCP Client Tools
writes prohibited
shell script for time-critical image storage use case 7-6 workflow example 3-83-9 URL encoding about 5-22 arcmkdir 6-4 arcmv 5-14 use cases image-file replication 7-27-4 time-critical image storage 7-47-6 username parameter, arcmv 5-16, 5-20 using HCP client tools 3-13-9 Python script for image-file replication use case 7-37-4 viewing arcfind logs 4-3 viewing arcfind output 4-3 workflow example 3-73-8 workflow examples 3-73-9 storage 3-43-7 tasks 3-43-6 WORM 1-2 writes prohibited 5-2
V
v parameter See also logs; verbose parameter arcfind 4-8 arcfind example 4-144-15 arcmkdir 6-4 arcmv 5-16 arcmv examples 5-295-30, 5-305-31 validate parameter 5-17 verbose parameter See also logs; v parameter about 2-5 arcfind 4-8 arcmkdir 6-4 arcmkdir example 6-6 arcmv 5-16 arcmv example 7-4 with loglevel parameter 5-26 version parameter about 2-5 arcfind 4-8 arcmkdir 6-4 arcmv 5-17 viewing arcfind logs in Windows 4-3 arcfind output in Windows 4-3
W
WARNING log level 5-13 WebDAV protocol about 2-2 arcmkdir support for 6-2 with arcmv 5-22 Windows arcfind examples 4-134-15 arcmkdir examples 6-56-6 arcmv examples 5-285-31 backslashes in directory paths 2-6
Index9
Using the HCP Client Tools
Index10
Using the HCP Client Tools
Hitachi Data Systems Corporate Headquarters 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 www.hds.com info@hds.com Asia Pacific and Americas 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 info@hds.com Europe Headquarters Sefton Park Stoke Poges Buckinghamshire SL2 4HD United Kingdom Phone: + 44 (0)1753 618000 info.eu@hds.com
MK-96ARC005-10