Debian Codegraph @

docs/source/index.rst

raw
1 azure-datalake-store
2 ====================
3
4 A pure-python interface to the Azure Data-lake Storage system, providing
5 pythonic file-system and file objects, seamless transition between Windows and
6 POSIX remote paths, high-performance up- and down-loader.
7
8 This software is under active development and not yet recommended for general
9 use.
10
11 Installation
12 ------------
13 Using ``pip``::
14
15 pip install azure-datalake-store
16
17 Manually (bleeding edge):
18
19 * Download the repo from https://github.com/Azure/azure-data-lake-store-python
20 * checkout the ``dev`` branch
21 * install the requirements (``pip install -r dev_requirements.txt``)
22 * install in develop mode (``python setup.py develop``)
23 * optionally: build the documentation (including this page) by running ``make html`` in the docs directory.
24
25
26 Auth
27 ----
28
29 Although users can generate and supply their own tokens to the base file-system
30 class, and there is a password-based function in the ``lib`` module for
31 generating tokens, the most convenient way to supply credentials is via
32 environment parameters. This latter method is the one used by default in
33 library. The following variables are required:
34
35 * azure_tenant_id
36 * azure_username
37 * azure_password
38 * azure_store_name
39 * azure_url_suffix (optional)
40
41 Pythonic Filesystem
42 -------------------
43
44 The ``AzureDLFileSystem`` object is the main API for library usage of this
45 package. It provides typical file-system operations on the remote azure
46 store
47
48 .. code-block:: python
49
50 token = lib.auth(tenant_id, username, password)
51 adl = core.AzureDLFileSystem(store_name, token)
52 # alternatively, adl = core.AzureDLFileSystem()
53 # uses environment variables
54
55 print(adl.ls()) # list files in the root directory
56 for item in adl.ls(detail=True):
57 print(item) # same, but with file details as dictionaries
58 print(adl.walk('')) # list all files at any directory depth
59 print('Usage:', adl.du('', deep=True, total=True)) # total bytes usage
60 adl.mkdir('newdir') # create directory
61 adl.touch('newdir/newfile') # create empty file
62 adl.put('remotefile', '/home/myuser/localfile') # upload a local file
63
64 In addition, the file-system generates file objects that are compatible with
65 the python file interface, ensuring compatibility with libraries that work on
66 python files. The recommended way to use this is with a context manager
67 (otherwise, be sure to call ``close()`` on the file object).
68
69 .. code-block:: python
70
71 with adl.open('newfile', 'wb') as f:
72 f.write(b'index,a,b\n')
73 f.tell() # now at position 9
74 f.flush() # forces data upstream
75 f.write(b'0,1,True')
76
77 with adl.open('newfile', 'rb') as f:
78 print(f.readlines())
79
80 with adl.open('newfile', 'rb') as f:
81 df = pd.read_csv(f) # read into pandas.
82
83 To seamlessly handle remote path representations across all supported platforms,
84 the main API will take in numerous path types: string, Path/PurePath, and
85 AzureDLPath. On Windows in particular, you can pass in paths separated by either
86 forward slashes or backslashes.
87
88 .. code-block:: python
89
90 import pathlib # only >= Python 3.4
91 from pathlib2 import pathlib # only <= Python 3.3
92
93 from azure.datalake.store.core import AzureDLPath
94
95 # possible remote paths to use on API
96 p1 = '\\foo\\bar'
97 p2 = '/foo/bar'
98 p3 = pathlib.PurePath('\\foo\\bar')
99 p4 = pathlib.PureWindowsPath('\\foo\\bar')
100 p5 = pathlib.PurePath('/foo/bar')
101 p6 = AzureDLPath('\\foo\\bar')
102 p7 = AzureDLPath('/foo/bar')
103
104 # p1, p3, and p6 only work on Windows
105 for p in [p1, p2, p3, p4, p5, p6, p7]:
106 with adl.open(p, 'rb') as f:
107 print(f.readlines())
108
109 Performant up-/down-loading
110 ---------------------------
111
112 Classes ``ADLUploader`` and ``ADLDownloader`` will chunk large files and send
113 many files to/from azure using multiple threads. A whole directory tree can
114 be transferred, files matching a specific glob-pattern or any particular file.
115
116 .. code-block:: python
117
118 # download the whole directory structure using 5 threads, 16MB chunks
119 ADLDownloader(adl, '', 'my_temp_dir', 5, 2**24)