use variable size chunks instead of always using max_seq_len

[sreeprasannar]

additions to the PretokDataset in relation to #15:

Keyword arguments:

• chunk_ratios: list[float] -- if set, batching is done with variable chunk sizes. The list of floats are used to calculate these variable chunk sizes by multiplying with the max_seq_len. Hence, the maximum ratio value (the max value for an element in this list of floats) can be 1.0. If chunk_ratios is specified, we expect to do variable size chunks sometimes less than the max_seq_len - this means we can pack more into a single batch. We calculate how much more based on the chunk size (directly proportional). If the chunk size is half of the max_seq_len then we can pack twice the number of batches

• batch_size_for_max_seq_len: int -- for chunks of size max_seq_len, this is the batch size. But if chunk_ratios are specified, we may adjust the batch size based on the chunk size (we'll pack more for smaller chunk sizes)

• max_shards - useful for quick debugging. Simply stop yielding after we reach max_shards number of shards