Metadata-Version: 1.1
Name: blobxfer
Version: 0.11.2
Summary: Azure storage transfer tool with AzCopy-like features
Home-page: https://github.com/Azure/blobxfer
Author: Microsoft Corporation, Azure Batch and HPC Team
Author-email: UNKNOWN
License: MIT
Description: .. image:: https://travis-ci.org/Azure/blobxfer.svg?branch=master
          :target: https://travis-ci.org/Azure/blobxfer
        .. image:: https://coveralls.io/repos/github/Azure/blobxfer/badge.svg?branch=master
          :target: https://coveralls.io/github/Azure/blobxfer?branch=master
        .. image:: https://img.shields.io/pypi/v/blobxfer.svg
          :target: https://pypi.python.org/pypi/blobxfer
        .. image:: https://img.shields.io/pypi/pyversions/blobxfer.svg
          :target: https://pypi.python.org/pypi/blobxfer
        .. image:: https://img.shields.io/pypi/l/blobxfer.svg
          :target: https://pypi.python.org/pypi/blobxfer
        
        blobxfer
        ========
        AzCopy-like OS independent Azure storage blob and file share transfer tool
        
        Installation
        ------------
        `blobxfer`_ is on PyPI and can be installed via:
        
        ::
        
          pip install blobxfer
        
        blobxfer is compatible with Python 2.7 and 3.3+. To install for Python 3, some
        distributions may use ``pip3`` instead.
        
        If you encounter difficulties installing the script, it may be due to the
        ``cryptography`` dependency. Please ensure that your system is able to install
        binary wheels provided by these dependencies (e.g., on Windows) or is able to
        compile the dependencies (i.e., ensure you have a C compiler, python, ssl,
        and ffi development libraries/headers installed prior to invoking pip). For
        instance, to install blobxfer on a fresh Ubuntu 14.04 installation for
        Python 2.7, issue the following commands:
        
        ::
        
            apt-get update
            apt-get install -y build-essential libssl-dev libffi-dev libpython-dev python-dev python-pip
            pip install --upgrade blobxfer
        
        If you need more fine-grained control on installing dependencies, continue
        reading this section. Depending upon the desired mode of authentication with
        Azure and options, the script will require the following packages, some of
        which will automatically pull required dependent packages. Below is a list of
        dependent packages:
        
        - Base Requirements
        
          - `azure-common`_ == 1.1.4
          - `azure-storage`_ == 0.32.0
          - `requests`_ == 2.10.0
        
        - Encryption Support
        
          - `cryptography`_ >= 1.4
        
        - Service Management Certificate Support
        
          - `azure-servicemanagement-legacy`_ == 0.20.3
        
        You can install these packages using pip, easy_install or through standard
        setup.py procedures. These dependencies will be automatically installed if
        using a package-based install or setup.py.
        
        .. _blobxfer: https://pypi.python.org/pypi/blobxfer
        .. _azure-common: https://pypi.python.org/pypi/azure-common
        .. _azure-storage: https://pypi.python.org/pypi/azure-storage
        .. _requests: https://pypi.python.org/pypi/requests
        .. _cryptography: https://pypi.python.org/pypi/cryptography
        .. _azure-servicemanagement-legacy: https://pypi.python.org/pypi/azure-servicemanagement-legacy
        
        Introduction
        ------------
        
        The blobxfer.py script allows interacting with storage accounts using any of
        the following methods: (1) management certificate, (2) shared account key,
        (3) SAS key. The script can, in addition to working with single files, mirror
        entire directories into and out of containers or file shares from Azure
        Storage, respectively. File and block/page level MD5 integrity checking is
        supported along with various transfer optimizations, built-in retries,
        user-specified timeouts, and client-side encryption.
        
        Program parameters and command-line options can be listed via the ``-h``
        switch. Please invoke this first if you are unfamiliar with blobxfer operation
        as not all options are explained below. At the minimum, three positional
        arguments are required: storage account name, container or share name, and
        local resource. Additionally, one of the following authentication switches
        must be supplied: ``--subscriptionid`` with ``--managementcert``,
        ``--storageaccountkey``, or ``--saskey``. Do not combine different
        authentication schemes together.
        
        Environment variables ``BLOBXFER_STORAGEACCOUNTKEY`` or ``BLOBXFER_SASKEY``
        can take the place of ``--storageaccountkey`` or ``--saskey``, respectively if
        you do not want to expose credentials on a command line.
        
        It is generally recommended to use SAS keys wherever appropriate; only HTTPS
        transport is used in the script. Please note that when using SAS keys that
        only container- or fileshare-level SAS keys will allow for entire directory
        uploading or container/fileshare downloading. The container/fileshare must
        also have been created beforehand, as containers/fileshares cannot be created
        using SAS keys.
        
        Example Usage
        -------------
        
        The following examples show how to invoke the script with commonly used
        options. Note that the authentication parameters are missing from the below
        examples. You will need to select a preferred method of authenticating with
        Azure and add the authentication switches (or as environment variables) as
        noted above.
        
        The script will attempt to perform a smart transfer, by detecting if the local
        resource exists. For example:
        
        ::
        
          blobxfer mystorageacct container0 mylocalfile.txt
        
        Note: if you downloaded the script directly from github, then you should append
        ``.py`` to the blobxfer command.
        
        If mylocalfile.txt exists locally, then the script will attempt to upload the
        file to container0 on mystorageacct. If the file does not exist, then it will
        attempt to download the resource. If the desired behavior is to download the
        file from Azure even if the local file exists, one can override the detection
        mechanism with ``--download``. ``--upload`` is available to force the transfer
        to Azure storage. Note that specifying a particular direction does not force
        the actual operation to occur as that depends on other options specified such
        as skipping on MD5 matches. Note that you may use the ``--remoteresource`` flag
        to rename the local file as the blob name on Azure storage if uploading,
        however, ``--remoteresource`` has no effect if uploading a directory of files.
        Please refer to the ``--collate`` option as explained below.
        
        If the local resource is a directory that exists, the script will attempt to
        mirror (recursively copy) the entire directory to Azure storage while
        maintaining subdirectories as virtual directories in Azure storage. You can
        disable the recursive copy (i.e., upload only the files in the directory)
        using the ``--no-recursive`` flag.
        
        To upload a directory with files only matching a Unix-style shell wildcard
        pattern, an example commandline would be:
        
        ::
        
          blobxfer mystorageacct container0 mylocaldir --upload --include '**/*.txt'
        
        This would attempt to recursively upload the contents of mylocaldir
        to container0 for any file matching the wildcard pattern ``*.txt`` within
        all subdirectories. Include patterns can be applied for uploads as well as
        downloads. Note that you will need to prevent globbing by your shell such
        that wildcard expansion does not take place before script interprets the
        argument.  If ``--include`` is not specified, all files will be uploaded
        or downloaded for the specific context.
        
        To download an entire container from your storage account, an example
        commandline would be:
        
        ::
        
          blobxfer mystorageacct container0 mylocaldir --remoteresource .
        
        Assuming mylocaldir directory does not exist, the script will attempt to
        download all of the contents in container0 because “.” is set with
        ``--remoteresource`` flag. To download individual blobs, one would specify the
        blob name instead of “.” with the ``--remoteresource`` flag. If mylocaldir
        directory exists, the script will attempt to upload the directory instead of
        downloading it. If you want to force the download direction even if the
        directory exists, indicate that with the ``--download`` flag. When downloading
        an entire container, the script will attempt to pre-allocate file space and
        recreate the sub-directory structure as needed.
        
        To collate files into specified virtual directories or local paths, use
        the ``--collate`` flag with the appropriate parameter. For example, the
        following commandline:
        
        ::
        
          blobxfer mystorageacct container0 myvhds --upload --collate vhds --autovhd
        
        If the directory ``myvhds`` had two vhd files a.vhd and subdir/b.vhd, these
        files would be uploaded into ``container0`` under the virtual directory named
        ``vhds``, and b.vhd would not contain the virtual directory subdir; thus,
        flattening the directory structure. The ``--autovhd`` flag would automatically
        enable page blob uploads for these files. If you wish to collate all files
        into the container directly, you would replace ``--collate vhds`` with
        ``--collate .``
        
        To strip leading components of a path on upload, use ``--strip-components``
        with a number argument which will act similarly to tar's
        ``--strip-components=NUMBER`` parameter. This parameter is only applied
        during an upload.
        
        To encrypt or decrypt files, the option ``--rsapublickey`` and
        ``--rsaprivatekey`` is available. This option requires a file location for a
        PEM encoded RSA public or private key. An optional parameter,
        ``--rsakeypassphrase`` is available for passphrase protected RSA private keys.
        
        To encrypt and upload, only the RSA public key is required although an RSA
        private key may be specified. To download and decrypt blobs which are
        encrypted, the RSA private key is required.
        
        ::
        
          blobxfer mystorageacct container0 myblobs --upload --rsapublickey mypublickey.pem
        
        The above example commandline would encrypt and upload files contained in
        ``myblobs`` using an RSA public key named ``mypublickey.pem``. An RSA private
        key may be specified instead for uploading (public parts will be used).
        
        ::
        
          blobxfer mystorageacct container0 myblobs --remoteresource . --download --rsaprivatekey myprivatekey.pem
        
        The above example commandline would download and decrypt all blobs in the
        container ``container0`` using an RSA private key named ``myprivatekey.pem``.
        An RSA private key must be specified for downloading and decryption of
        encrypted blobs.
        
        Currently only the ``FullBlob`` encryption mode is supported for the
        parameter ``--encmode``. The ``FullBlob`` encryption mode either uploads or
        downloads Azure Storage .NET/Java compatible client-side encrypted block blobs.
        
        Please read important points in the Encryption Notes below for more
        information.
        
        To transfer to an Azure Files share, specify the ``--fileshare`` option and
        specify the share name as the second positional argument.
        
        ::
        
          blobxfer mystorageacct myshare localfiles --fileshare --upload
        
        The above example would upload all files in the ``localfiles`` directory to
        the share named ``myshare``. Encryption/decryption options are compatible with
        Azure Files as the destination or source. Please refer to this `MSDN article`_
        for features not supported by the Azure File Service.
        
        .. _MSDN article: https://msdn.microsoft.com/en-us/library/azure/dn744326.aspx
        
        General Notes
        -------------
        
        - blobxfer does not take any leases on blobs or containers. It is up to
          the user to ensure that blobs are not modified while download/uploads
          are being performed.
        - No validation is performed regarding container and file naming and length
          restrictions.
        - blobxfer will attempt to download from blob storage as-is. If the source
          filename is incompatible with the destination operating system, then
          failure may result.
        - When using SAS, the SAS key must be a container-level SAS if performing
          recursive directory upload or container download.
        - If uploading via SAS, the container must already be created in blob
          storage prior to upload. This is a limitation of SAS keys. The script
          will force disable container creation if a SAS key is specified.
        - For non-SAS requests, timeouts may not be properly honored due to
          limitations of the Azure Python SDK.
        - By default, files with matching MD5 checksums will be skipped for both
          download (if MD5 information is present on the blob) and upload. Specify
          ``--no-skiponmatch`` to disable this functionality.
        - When uploading files as page blobs, the content is page boundary
          byte-aligned. The MD5 for the blob is computed using the final aligned
          data if the source is not page boundary byte-aligned. This enables these
          page blobs or files to be skipped during subsequent download or upload,
          if the ``--no-skiponmatch`` parameter is not specified.
        - If ``--delete`` is specified, any remote files found that have no
          corresponding local file in directory upload mode will be deleted. Deletion
          occurs prior to any transfers, analogous to the delete-before rsync option.
          Please note that this parameter will interact with ``--include`` and any
          file not included from the include pattern will be deleted.
        - ``--include`` has no effect when specifying a single file to upload or
          blob to download. When specifying ``--include`` on container download,
          the pattern will be applied to the blob name without the container name.
          Globbing of wildcards must be disabled such that the script can read
          the include pattern without the shell expanding the wildcards, if specified.
        - Empty directories are not created locally when downloading from an Azure
          file share which has empty directories.
        - Empty directories are not deleted if ``--delete`` is specified and no
          files remain in the directory on the Azure file share.
        
        Performance Notes
        -----------------
        
        - Most likely, you will need to tweak the ``--numworkers`` argument that best
          suits your environment. The default is the number of CPUs on the running
          machine multiplied by 3 (except when transferring to/from file shares).
          Increasing this number (or even using the default) may not provide the
          optimal balance between concurrency and your network conditions.
          Additionally, this number may not work properly if you are attempting to
          run multiple blobxfer sessions in parallel from one machine or IP address.
          Futhermore, this number may be defaulted to be set too high if encryption
          is enabled and the machine cannot handle processing multiple threads in
          parallel.
        - Computing file MD5 can be time consuming for large files. If integrity
          checking or rsync-like capability is not required, specify
          ``--no-computefilemd5`` to disable MD5 computation for files.
        - File share performance can be "slow" or become a bottleneck, especially for
          file shares containing thousands of files as multiple REST calls must be
          performed for each file. Currently, a single file share has a limit of up
          to 60 MB/s and 1000 8KB IOPS. Please refer to the
          `Azure Storage Scalability and Performance Targets`_ for performance targets
          and limits regarding Azure Storage Blobs and Files. If scalable high
          performance is required, consider using blob storage or multiple file
          shares.
        - Using SAS keys may provide the best performance as the script bypasses
          the Azure Storage Python SDK and uses requests/urllib3 directly with
          Azure Storage endpoints. Transfers to/from Azure Files will always use
          the Azure Storage Python SDK even with SAS keys.
        - As of requests 2.6.0 and Python versions < 2.7.9 (i.e., interpreter found
          on default Ubuntu 14.04 installations), if certain packages are installed,
          as those found in ``requests[security]`` then the underlying ``urllib3``
          package will utilize the ``ndg-httpsclient`` package which will use
          `pyOpenSSL`_. This will ensure the peers are `fully validated`_. However,
          this incurs a rather larger performance penalty. If you understand the
          potential security risks for disabling this behavior due to high performance
          requirements, you can either remove ``ndg-httpsclient`` or use the script
          in a ``virtualenv`` environment without the ``ndg-httpsclient`` package.
          Python versions >= 2.7.9 are not affected by this issue. These warnings can
          be suppressed using ``--disable-urllib-warnings``, but is not recommended
          unless you understand the security implications.
        
        .. _Azure Storage Scalability and Performance Targets: https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/
        .. _pyOpenSSL: https://urllib3.readthedocs.org/en/latest/security.html#pyopenssl
        .. _fully validated: https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
        
        
        Encryption Notes
        ----------------
        
        - All required information regarding the encryption process is stored on
          each blob's ``encryptiondata`` and ``encryptiondata_authentication``
          metadata. These metadata entries are used on download to configure the proper
          download and parameters for the decryption process as well as to authenticate
          the encryption. Encryption metadata set by blobxfer (or the Azure Storage
          .NET/Java client library) should not be modified or blobs may be
          unrecoverable.
        - Local files can be encrypted by blobxfer and stored in Azure Files and,
          correspondingly, remote files on Azure File shares can be decrypted by
          blobxfer as long as the metdata portions remain in-tact.
        - Keys for AES256 block cipher are generated on a per-blob/file basis. These
          keys are encrypted using RSAES-OAEP.
        - MD5 for both the pre-encrypted and encrypted version of the file is stored
          in blob/file metadata. Rsync-like synchronization is still supported
          transparently with encrypted blobs/files.
        - Whole file MD5 checks are skipped if a message authentication code is found
          to validate the integrity of the encrypted data.
        - Attempting to upload the same file as an encrypted blob with a different RSA
          key or under a different encryption mode will not occur if the file content
          MD5 is the same. This behavior can be overridden by including the option
          ``--no-skiponmatch``.
        - If one wishes to apply encryption to a blob/file already uploaded to Azure
          Storage that has not changed, the upload will not occur since the underlying
          file content MD5 has not changed; this behavior can be overriden by
          including the option ``--no-skiponmatch``.
        - Encryption is only applied to block blobs (or fileshare files). Encrypted
          page blobs appear to be of minimal value stored in Azure Storage via
          blobxfer. Thus, if uploading VHDs while enabling encryption in the script,
          do not enable the option ``--pageblob``. ``--autovhd`` will continue to work
          transparently where vhd files will be uploaded as page blobs in unencrypted
          form while other files will be uploaded as encrypted block blobs. Note that
          using ``--autovhd`` with encryption will force set the max chunk size to
          4 MiB for non-encrypted vhd files.
        - Downloading encrypted blobs/files may not fully preallocate each file due to
          padding. Script failure can result during transfer if there is insufficient
          disk space.
        - Zero-byte (empty) files are not encrypted.
        
        Change Log
        ----------
        
        See the `CHANGELOG.md`_ file.
        
        .. _CHANGELOG.md: https://github.com/Azure/blobxfer/CHANGELOG.md
Keywords: azcopy azure storage blob files transfer copy smb
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Topic :: Utilities
