Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow set max_operations in meta class for batch writes #598

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

DKbyo
Copy link

@DKbyo DKbyo commented Feb 27, 2019

No description provided.

@ikonst
Copy link
Contributor

ikonst commented Feb 28, 2019

Could you explain the rationale? Per DynamoDB docs:

A single BatchWriteItem operation can contain up to 25 PutItem or DeleteItem requests. The total size of all the items written cannot exceed 16 MB.

Do you want to set it to less than 25? Also, wouldn't it useful to be configurable as a parameter to the batch_write method rather than a single value per model?

@DKbyo
Copy link
Author

DKbyo commented Feb 28, 2019

Well, if you have capacity units limited for small tables/projects, you need to limit batch writes and put some delay between them. And yes... will be more useful to set it as a parameter in the batch_write method if you change the capacity units dinamically.

@ikonst
Copy link
Contributor

ikonst commented Mar 1, 2019

BatchWrite doesn't perform any throttling though (i.e. there's no time.sleep or anything like that). Let's say you reduce the number from 25 to 15, you'd simply be making HTTP calls more often, but your capacity utilization would still be the same.

From the docs:

Parallel processing reduces latency, but each specified put and delete request consumes the same number of write capacity units whether it is processed in parallel or not.

Would this work for your case?

from more_itertools import chunked

batch_writer = MyModel.batch_write()
for models_chunk in chunked(models, 15):
   for model in models_chunk:
      batch_writer.save(model)
   batch_writer.commit()
   time.sleep(1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants