| 1 |
azure-datalake-store |
| 2 |
==================== |
| 3 |
|
| 4 |
A pure-python interface to the Azure Data-lake Storage system, providing |
| 5 |
pythonic file-system and file objects, seamless transition between Windows and |
| 6 |
POSIX remote paths, high-performance up- and down-loader. |
| 7 |
|
| 8 |
This software is under active development and not yet recommended for general |
| 9 |
use. |
| 10 |
|
| 11 |
Installation |
| 12 |
------------ |
| 13 |
Using ``pip``:: |
| 14 |
|
| 15 |
pip install azure-datalake-store |
| 16 |
|
| 17 |
Manually (bleeding edge): |
| 18 |
|
| 19 |
* Download the repo from https://github.com/Azure/azure-data-lake-store-python |
| 20 |
* checkout the ``dev`` branch |
| 21 |
* install the requirements (``pip install -r dev_requirements.txt``) |
| 22 |
* install in develop mode (``python setup.py develop``) |
| 23 |
* optionally: build the documentation (including this page) by running ``make html`` in the docs directory. |
| 24 |
|
| 25 |
|
| 26 |
Auth |
| 27 |
---- |
| 28 |
|
| 29 |
Although users can generate and supply their own tokens to the base file-system |
| 30 |
class, and there is a password-based function in the ``lib`` module for |
| 31 |
generating tokens, the most convenient way to supply credentials is via |
| 32 |
environment parameters. This latter method is the one used by default in |
| 33 |
library. The following variables are required: |
| 34 |
|
| 35 |
* azure_tenant_id |
| 36 |
* azure_username |
| 37 |
* azure_password |
| 38 |
* azure_store_name |
| 39 |
* azure_url_suffix (optional) |
| 40 |
|
| 41 |
Pythonic Filesystem |
| 42 |
------------------- |
| 43 |
|
| 44 |
The ``AzureDLFileSystem`` object is the main API for library usage of this |
| 45 |
package. It provides typical file-system operations on the remote azure |
| 46 |
store |
| 47 |
|
| 48 |
.. code-block:: python |
| 49 |
|
| 50 |
token = lib.auth(tenant_id, username, password) |
| 51 |
adl = core.AzureDLFileSystem(store_name, token) |
| 52 |
# alternatively, adl = core.AzureDLFileSystem() |
| 53 |
# uses environment variables |
| 54 |
|
| 55 |
print(adl.ls()) # list files in the root directory |
| 56 |
for item in adl.ls(detail=True): |
| 57 |
print(item) # same, but with file details as dictionaries |
| 58 |
print(adl.walk('')) # list all files at any directory depth |
| 59 |
print('Usage:', adl.du('', deep=True, total=True)) # total bytes usage |
| 60 |
adl.mkdir('newdir') # create directory |
| 61 |
adl.touch('newdir/newfile') # create empty file |
| 62 |
adl.put('remotefile', '/home/myuser/localfile') # upload a local file |
| 63 |
|
| 64 |
In addition, the file-system generates file objects that are compatible with |
| 65 |
the python file interface, ensuring compatibility with libraries that work on |
| 66 |
python files. The recommended way to use this is with a context manager |
| 67 |
(otherwise, be sure to call ``close()`` on the file object). |
| 68 |
|
| 69 |
.. code-block:: python |
| 70 |
|
| 71 |
with adl.open('newfile', 'wb') as f: |
| 72 |
f.write(b'index,a,b\n') |
| 73 |
f.tell() # now at position 9 |
| 74 |
f.flush() # forces data upstream |
| 75 |
f.write(b'0,1,True') |
| 76 |
|
| 77 |
with adl.open('newfile', 'rb') as f: |
| 78 |
print(f.readlines()) |
| 79 |
|
| 80 |
with adl.open('newfile', 'rb') as f: |
| 81 |
df = pd.read_csv(f) # read into pandas. |
| 82 |
|
| 83 |
To seamlessly handle remote path representations across all supported platforms, |
| 84 |
the main API will take in numerous path types: string, Path/PurePath, and |
| 85 |
AzureDLPath. On Windows in particular, you can pass in paths separated by either |
| 86 |
forward slashes or backslashes. |
| 87 |
|
| 88 |
.. code-block:: python |
| 89 |
|
| 90 |
import pathlib # only >= Python 3.4 |
| 91 |
from pathlib2 import pathlib # only <= Python 3.3 |
| 92 |
|
| 93 |
from azure.datalake.store.core import AzureDLPath |
| 94 |
|
| 95 |
# possible remote paths to use on API |
| 96 |
p1 = '\\foo\\bar' |
| 97 |
p2 = '/foo/bar' |
| 98 |
p3 = pathlib.PurePath('\\foo\\bar') |
| 99 |
p4 = pathlib.PureWindowsPath('\\foo\\bar') |
| 100 |
p5 = pathlib.PurePath('/foo/bar') |
| 101 |
p6 = AzureDLPath('\\foo\\bar') |
| 102 |
p7 = AzureDLPath('/foo/bar') |
| 103 |
|
| 104 |
# p1, p3, and p6 only work on Windows |
| 105 |
for p in [p1, p2, p3, p4, p5, p6, p7]: |
| 106 |
with adl.open(p, 'rb') as f: |
| 107 |
print(f.readlines()) |
| 108 |
|
| 109 |
Performant up-/down-loading |
| 110 |
--------------------------- |
| 111 |
|
| 112 |
Classes ``ADLUploader`` and ``ADLDownloader`` will chunk large files and send |
| 113 |
many files to/from azure using multiple threads. A whole directory tree can |
| 114 |
be transferred, files matching a specific glob-pattern or any particular file. |
| 115 |
|
| 116 |
.. code-block:: python |
| 117 |
|
| 118 |
# download the whole directory structure using 5 threads, 16MB chunks |
| 119 |
ADLDownloader(adl, '', 'my_temp_dir', 5, 2**24) |