| 1 |
.. _dynamodb_guide: |
| 2 |
|
| 3 |
Amazon DynamoDB |
| 4 |
================ |
| 5 |
By following this guide, you will learn how to use the |
| 6 |
:py:class:`DynamoDB.ServiceResource` and :py:class:`DynamoDB.Table` |
| 7 |
resources in order to create tables, write items to tables, modify existing |
| 8 |
items, retrieve items, and query/filter the items in the table. |
| 9 |
|
| 10 |
|
| 11 |
Creating a new table |
| 12 |
-------------------- |
| 13 |
|
| 14 |
In order to create a new table, use the |
| 15 |
:py:meth:`DynamoDB.ServiceResource.create_table` method:: |
| 16 |
|
| 17 |
import boto3 |
| 18 |
|
| 19 |
# Get the service resource. |
| 20 |
dynamodb = boto3.resource('dynamodb') |
| 21 |
|
| 22 |
# Create the DynamoDB table. |
| 23 |
table = dynamodb.create_table( |
| 24 |
TableName='users', |
| 25 |
KeySchema=[ |
| 26 |
{ |
| 27 |
'AttributeName': 'username', |
| 28 |
'KeyType': 'HASH' |
| 29 |
}, |
| 30 |
{ |
| 31 |
'AttributeName': 'last_name', |
| 32 |
'KeyType': 'RANGE' |
| 33 |
} |
| 34 |
], |
| 35 |
AttributeDefinitions=[ |
| 36 |
{ |
| 37 |
'AttributeName': 'username', |
| 38 |
'AttributeType': 'S' |
| 39 |
}, |
| 40 |
{ |
| 41 |
'AttributeName': 'last_name', |
| 42 |
'AttributeType': 'S' |
| 43 |
}, |
| 44 |
], |
| 45 |
ProvisionedThroughput={ |
| 46 |
'ReadCapacityUnits': 5, |
| 47 |
'WriteCapacityUnits': 5 |
| 48 |
} |
| 49 |
) |
| 50 |
|
| 51 |
# Wait until the table exists. |
| 52 |
table.wait_until_exists() |
| 53 |
|
| 54 |
# Print out some data about the table. |
| 55 |
print(table.item_count) |
| 56 |
|
| 57 |
Expected output:: |
| 58 |
|
| 59 |
0 |
| 60 |
|
| 61 |
This creates a table named ``users`` that respectively has the hash and |
| 62 |
range primary keys ``username`` and ``last_name``. |
| 63 |
This method will return a :py:class:`DynamoDB.Table` resource to call |
| 64 |
additional methods on the created table. |
| 65 |
|
| 66 |
|
| 67 |
Using an existing table |
| 68 |
----------------------- |
| 69 |
It is also possible to create a :py:class:`DynamoDB.Table` resource from |
| 70 |
an existing table:: |
| 71 |
|
| 72 |
import boto3 |
| 73 |
|
| 74 |
# Get the service resource. |
| 75 |
dynamodb = boto3.resource('dynamodb') |
| 76 |
|
| 77 |
# Instantiate a table resource object without actually |
| 78 |
# creating a DynamoDB table. Note that the attributes of this table |
| 79 |
# are lazy-loaded: a request is not made nor are the attribute |
| 80 |
# values populated until the attributes |
| 81 |
# on the table resource are accessed or its load() method is called. |
| 82 |
table = dynamodb.Table('users') |
| 83 |
|
| 84 |
# Print out some data about the table. |
| 85 |
# This will cause a request to be made to DynamoDB and its attribute |
| 86 |
# values will be set based on the response. |
| 87 |
print(table.creation_date_time) |
| 88 |
|
| 89 |
Expected output (Please note that the actual times will probably not match up):: |
| 90 |
|
| 91 |
2015-06-26 12:42:45.149000-07:00 |
| 92 |
|
| 93 |
|
| 94 |
Creating a new item |
| 95 |
------------------- |
| 96 |
|
| 97 |
Once you have a :py:class:`DynamoDB.Table` resource you can add new items |
| 98 |
to the table using :py:meth:`DynamoDB.Table.put_item`:: |
| 99 |
|
| 100 |
table.put_item( |
| 101 |
Item={ |
| 102 |
'username': 'janedoe', |
| 103 |
'first_name': 'Jane', |
| 104 |
'last_name': 'Doe', |
| 105 |
'age': 25, |
| 106 |
'account_type': 'standard_user', |
| 107 |
} |
| 108 |
) |
| 109 |
|
| 110 |
For all of the valid types that can be used for an item, refer to |
| 111 |
:ref:`ref_valid_dynamodb_types`. |
| 112 |
|
| 113 |
|
| 114 |
Getting an item |
| 115 |
--------------- |
| 116 |
You can then retrieve the object using :py:meth:`DynamoDB.Table.get_item`:: |
| 117 |
|
| 118 |
response = table.get_item( |
| 119 |
Key={ |
| 120 |
'username': 'janedoe', |
| 121 |
'last_name': 'Doe' |
| 122 |
} |
| 123 |
) |
| 124 |
item = response['Item'] |
| 125 |
print(item) |
| 126 |
|
| 127 |
|
| 128 |
Expected output:: |
| 129 |
|
| 130 |
{u'username': u'janedoe', |
| 131 |
u'first_name': u'Jane', |
| 132 |
u'last_name': u'Doe', |
| 133 |
u'account_type': u'standard_user', |
| 134 |
u'age': Decimal('25')} |
| 135 |
|
| 136 |
|
| 137 |
Updating an item |
| 138 |
---------------- |
| 139 |
|
| 140 |
You can then update attributes of the item in the table |
| 141 |
using :py:meth:`DynamoDB.Table.update_item`:: |
| 142 |
|
| 143 |
table.update_item( |
| 144 |
Key={ |
| 145 |
'username': 'janedoe', |
| 146 |
'last_name': 'Doe' |
| 147 |
}, |
| 148 |
UpdateExpression='SET age = :val1', |
| 149 |
ExpressionAttributeValues={ |
| 150 |
':val1': 26 |
| 151 |
} |
| 152 |
) |
| 153 |
|
| 154 |
Then if you retrieve the item again, it will be updated appropriately:: |
| 155 |
|
| 156 |
response = table.get_item( |
| 157 |
Key={ |
| 158 |
'username': 'janedoe', |
| 159 |
'last_name': 'Doe' |
| 160 |
} |
| 161 |
) |
| 162 |
item = response['Item'] |
| 163 |
print(item) |
| 164 |
|
| 165 |
|
| 166 |
Expected output:: |
| 167 |
|
| 168 |
{u'username': u'janedoe', |
| 169 |
u'first_name': u'Jane', |
| 170 |
u'last_name': u'Doe', |
| 171 |
u'account_type': u'standard_user', |
| 172 |
u'age': Decimal('26')} |
| 173 |
|
| 174 |
|
| 175 |
Deleting an item |
| 176 |
---------------- |
| 177 |
You can also delete the item using :py:meth:`DynamoDB.Table.delete_item`:: |
| 178 |
|
| 179 |
table.delete_item( |
| 180 |
Key={ |
| 181 |
'username': 'janedoe', |
| 182 |
'last_name': 'Doe' |
| 183 |
} |
| 184 |
) |
| 185 |
|
| 186 |
|
| 187 |
Batch writing |
| 188 |
------------- |
| 189 |
If you are loading a lot of data at a time, you can make use of |
| 190 |
:py:meth:`DynamoDB.Table.batch_writer` so you can both speed up the process and |
| 191 |
reduce the number of write requests made to the service. |
| 192 |
|
| 193 |
This method returns a handle to a batch writer object that will automatically |
| 194 |
handle buffering and sending items in batches. In addition, the |
| 195 |
batch writer will also automatically handle any unprocessed items and |
| 196 |
resend them as needed. All you need to do is call ``put_item`` for any |
| 197 |
items you want to add, and ``delete_item`` for any items you want to delete:: |
| 198 |
|
| 199 |
with table.batch_writer() as batch: |
| 200 |
batch.put_item( |
| 201 |
Item={ |
| 202 |
'account_type': 'standard_user', |
| 203 |
'username': 'johndoe', |
| 204 |
'first_name': 'John', |
| 205 |
'last_name': 'Doe', |
| 206 |
'age': 25, |
| 207 |
'address': { |
| 208 |
'road': '1 Jefferson Street', |
| 209 |
'city': 'Los Angeles', |
| 210 |
'state': 'CA', |
| 211 |
'zipcode': 90001 |
| 212 |
} |
| 213 |
} |
| 214 |
) |
| 215 |
batch.put_item( |
| 216 |
Item={ |
| 217 |
'account_type': 'super_user', |
| 218 |
'username': 'janedoering', |
| 219 |
'first_name': 'Jane', |
| 220 |
'last_name': 'Doering', |
| 221 |
'age': 40, |
| 222 |
'address': { |
| 223 |
'road': '2 Washington Avenue', |
| 224 |
'city': 'Seattle', |
| 225 |
'state': 'WA', |
| 226 |
'zipcode': 98109 |
| 227 |
} |
| 228 |
} |
| 229 |
) |
| 230 |
batch.put_item( |
| 231 |
Item={ |
| 232 |
'account_type': 'standard_user', |
| 233 |
'username': 'bobsmith', |
| 234 |
'first_name': 'Bob', |
| 235 |
'last_name': 'Smith', |
| 236 |
'age': 18, |
| 237 |
'address': { |
| 238 |
'road': '3 Madison Lane', |
| 239 |
'city': 'Louisville', |
| 240 |
'state': 'KY', |
| 241 |
'zipcode': 40213 |
| 242 |
} |
| 243 |
} |
| 244 |
) |
| 245 |
batch.put_item( |
| 246 |
Item={ |
| 247 |
'account_type': 'super_user', |
| 248 |
'username': 'alicedoe', |
| 249 |
'first_name': 'Alice', |
| 250 |
'last_name': 'Doe', |
| 251 |
'age': 27, |
| 252 |
'address': { |
| 253 |
'road': '1 Jefferson Street', |
| 254 |
'city': 'Los Angeles', |
| 255 |
'state': 'CA', |
| 256 |
'zipcode': 90001 |
| 257 |
} |
| 258 |
} |
| 259 |
) |
| 260 |
|
| 261 |
The batch writer is even able to handle a very large amount of writes to the |
| 262 |
table. |
| 263 |
|
| 264 |
:: |
| 265 |
|
| 266 |
with table.batch_writer() as batch: |
| 267 |
for i in range(50): |
| 268 |
batch.put_item( |
| 269 |
Item={ |
| 270 |
'account_type': 'anonymous', |
| 271 |
'username': 'user' + str(i), |
| 272 |
'first_name': 'unknown', |
| 273 |
'last_name': 'unknown' |
| 274 |
} |
| 275 |
) |
| 276 |
|
| 277 |
The batch writer can help to de-duplicate request by specifying ``overwrite_by_pkeys=['partition_key', 'sort_key']`` |
| 278 |
if you want to bypass no duplication limitation of single batch write request as |
| 279 |
``botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: Provided list of item keys contains duplicates``. |
| 280 |
|
| 281 |
It will drop request items in the buffer if their primary keys(composite) values are |
| 282 |
the same as newly added one, as eventually consistent with streams of individual |
| 283 |
put/delete operations on the same item. |
| 284 |
|
| 285 |
:: |
| 286 |
|
| 287 |
with table.batch_writer(overwrite_by_pkeys=['partition_key', 'sort_key']) as batch: |
| 288 |
batch.put_item( |
| 289 |
Item={ |
| 290 |
'partition_key': 'p1', |
| 291 |
'sort_key': 's1', |
| 292 |
'other': '111', |
| 293 |
} |
| 294 |
) |
| 295 |
batch.put_item( |
| 296 |
Item={ |
| 297 |
'partition_key': 'p1', |
| 298 |
'sort_key': 's1', |
| 299 |
'other': '222', |
| 300 |
} |
| 301 |
) |
| 302 |
batch.delete_item( |
| 303 |
Key={ |
| 304 |
'partition_key': 'p1', |
| 305 |
'sort_key': 's2' |
| 306 |
} |
| 307 |
) |
| 308 |
batch.put_item( |
| 309 |
Item={ |
| 310 |
'partition_key': 'p1', |
| 311 |
'sort_key': 's2', |
| 312 |
'other': '444', |
| 313 |
} |
| 314 |
) |
| 315 |
|
| 316 |
after de-duplicate: |
| 317 |
|
| 318 |
:: |
| 319 |
|
| 320 |
batch.put_item( |
| 321 |
Item={ |
| 322 |
'partition_key': 'p1', |
| 323 |
'sort_key': 's1', |
| 324 |
'other': '222', |
| 325 |
} |
| 326 |
) |
| 327 |
batch.put_item( |
| 328 |
Item={ |
| 329 |
'partition_key': 'p1', |
| 330 |
'sort_key': 's2', |
| 331 |
'other': '444', |
| 332 |
} |
| 333 |
) |
| 334 |
|
| 335 |
|
| 336 |
Querying and scanning |
| 337 |
--------------------- |
| 338 |
|
| 339 |
With the table full of items, you can then query or scan the items in the table |
| 340 |
using the :py:meth:`DynamoDB.Table.query` or :py:meth:`DynamoDB.Table.scan` |
| 341 |
methods respectively. To add conditions to scanning and querying the table, |
| 342 |
you will need to import the :py:class:`boto3.dynamodb.conditions.Key` and |
| 343 |
:py:class:`boto3.dynamodb.conditions.Attr` classes. The |
| 344 |
:py:class:`boto3.dynamodb.conditions.Key` should be used when the |
| 345 |
condition is related to the key of the item. |
| 346 |
The :py:class:`boto3.dynamodb.conditions.Attr` should be used when the |
| 347 |
condition is related to an attribute of the item:: |
| 348 |
|
| 349 |
from boto3.dynamodb.conditions import Key, Attr |
| 350 |
|
| 351 |
|
| 352 |
This queries for all of the users whose ``username`` key equals ``johndoe``:: |
| 353 |
|
| 354 |
response = table.query( |
| 355 |
KeyConditionExpression=Key('username').eq('johndoe') |
| 356 |
) |
| 357 |
items = response['Items'] |
| 358 |
print(items) |
| 359 |
|
| 360 |
|
| 361 |
Expected output:: |
| 362 |
|
| 363 |
[{u'username': u'johndoe', |
| 364 |
u'first_name': u'John', |
| 365 |
u'last_name': u'Doe', |
| 366 |
u'account_type': u'standard_user', |
| 367 |
u'age': Decimal('25'), |
| 368 |
u'address': {u'city': u'Los Angeles', |
| 369 |
u'state': u'CA', |
| 370 |
u'zipcode': Decimal('90001'), |
| 371 |
u'road': u'1 Jefferson Street'}}] |
| 372 |
|
| 373 |
|
| 374 |
Similarly you can scan the table based on attributes of the items. For |
| 375 |
example, this scans for all the users whose ``age`` is less than ``27``:: |
| 376 |
|
| 377 |
response = table.scan( |
| 378 |
FilterExpression=Attr('age').lt(27) |
| 379 |
) |
| 380 |
items = response['Items'] |
| 381 |
print(items) |
| 382 |
|
| 383 |
|
| 384 |
Expected output:: |
| 385 |
|
| 386 |
[{u'username': u'johndoe', |
| 387 |
u'first_name': u'John', |
| 388 |
u'last_name': u'Doe', |
| 389 |
u'account_type': u'standard_user', |
| 390 |
u'age': Decimal('25'), |
| 391 |
u'address': {u'city': u'Los Angeles', |
| 392 |
u'state': u'CA', |
| 393 |
u'zipcode': Decimal('90001'), |
| 394 |
u'road': u'1 Jefferson Street'}}, |
| 395 |
{u'username': u'bobsmith', |
| 396 |
u'first_name': u'Bob', |
| 397 |
u'last_name': u'Smith', |
| 398 |
u'account_type': u'standard_user', |
| 399 |
u'age': Decimal('18'), |
| 400 |
u'address': {u'city': u'Louisville', |
| 401 |
u'state': u'KY', |
| 402 |
u'zipcode': Decimal('40213'), |
| 403 |
u'road': u'3 Madison Lane'}}] |
| 404 |
|
| 405 |
|
| 406 |
You are also able to chain conditions together using the logical operators: |
| 407 |
``&`` (and), ``|`` (or), and ``~`` (not). For example, this scans for all |
| 408 |
users whose ``first_name`` starts with ``J`` and whose ``account_type`` is |
| 409 |
``super_user``:: |
| 410 |
|
| 411 |
response = table.scan( |
| 412 |
FilterExpression=Attr('first_name').begins_with('J') & Attr('account_type').eq('super_user') |
| 413 |
) |
| 414 |
items = response['Items'] |
| 415 |
print(items) |
| 416 |
|
| 417 |
|
| 418 |
Expected output:: |
| 419 |
|
| 420 |
[{u'username': u'janedoering', |
| 421 |
u'first_name': u'Jane', |
| 422 |
u'last_name': u'Doering', |
| 423 |
u'account_type': u'super_user', |
| 424 |
u'age': Decimal('40'), |
| 425 |
u'address': {u'city': u'Seattle', |
| 426 |
u'state': u'WA', |
| 427 |
u'zipcode': Decimal('98109'), |
| 428 |
u'road': u'2 Washington Avenue'}}] |
| 429 |
|
| 430 |
|
| 431 |
You can even scan based on conditions of a nested attribute. For example this |
| 432 |
scans for all users whose ``state`` in their ``address`` is ``CA``:: |
| 433 |
|
| 434 |
response = table.scan( |
| 435 |
FilterExpression=Attr('address.state').eq('CA') |
| 436 |
) |
| 437 |
items = response['Items'] |
| 438 |
print(items) |
| 439 |
|
| 440 |
|
| 441 |
Expected output:: |
| 442 |
|
| 443 |
[{u'username': u'johndoe', |
| 444 |
u'first_name': u'John', |
| 445 |
u'last_name': u'Doe', |
| 446 |
u'account_type': u'standard_user', |
| 447 |
u'age': Decimal('25'), |
| 448 |
u'address': {u'city': u'Los Angeles', |
| 449 |
u'state': u'CA', |
| 450 |
u'zipcode': Decimal('90001'), |
| 451 |
u'road': u'1 Jefferson Street'}}, |
| 452 |
{u'username': u'alicedoe', |
| 453 |
u'first_name': u'Alice', |
| 454 |
u'last_name': u'Doe', |
| 455 |
u'account_type': u'super_user', |
| 456 |
u'age': Decimal('27'), |
| 457 |
u'address': {u'city': u'Los Angeles', |
| 458 |
u'state': u'CA', |
| 459 |
u'zipcode': Decimal('90001'), |
| 460 |
u'road': u'1 Jefferson Street'}}] |
| 461 |
|
| 462 |
|
| 463 |
For more information on the various conditions you can use for queries and |
| 464 |
scans, refer to :ref:`ref_dynamodb_conditions`. |
| 465 |
|
| 466 |
|
| 467 |
Deleting a table |
| 468 |
---------------- |
| 469 |
Finally, if you want to delete your table call |
| 470 |
:py:meth:`DynamoDB.Table.delete`:: |
| 471 |
|
| 472 |
table.delete() |