upload files > 10 MB in chunks

Author: CalebEverett

CHANGELOG.md

@@ -5,7 +5,12 @@ All notable changes starting with v0.1.34 to this project will be documented in
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-# v0.1.55 (2022-01-29
+# v0.1.56 (2022-02-07)
+- **changed:** increase default bundle size to 100 MB to reduce number of tx and use `chunk/` endpoint instead of `tx/` endpoint used for bundles 10 MB or less.
+- **changed:** post transactions in chunks when uploading files greater than 10 MB to allow individual files of unlimited size to be uploaded.
+- **added:** include [upload_files](examples/upload_files.rs) example.
+
+# v0.1.55 (2022-01-29)
 - **added:** include `write-metaplex-items` in `upload-nfts` command.
 
 # v0.1.54 (2022-01-18)

Cargo.toml

@@ -2,7 +2,7 @@
 name = "arloader"
 authors = ["calebeverett <[email protected]>"]
 description = "Command line application and library for uploading files to Arweave."
-version = "0.1.55"
+version = "0.1.56"
 edition = "2021"
 license = "Apache-2.0"
 repository = "https://github.com/CalebEverett/arloader"

README.md

@@ -20,7 +20,6 @@ Upload gigabytes of files with one command. Files are read and posted to [arweav
 * [Pricing Comparison](#pricing-comparison)
 * [Roadmap](#roadmap)
 * [Transactions Prior to v0.1.51](#potential-issue-with-transactions-uploaded-prior-to-version-0.1.51)
-* [Tokens of Appreciation](#tokens-of-appreciation)
 
 ## Installation
 
@@ -40,7 +39,7 @@ cargo install arloader
 4. If you're going to use SOL, get a [Solana wallet](https://docs.solana.com/wallet-guide/cli) json file and transfer some SOL to it.
 
 ## NFT Usage
-The single `upload-nfts` command below assumes you have a pair of image and metadata files for each of your NFTs. To learn more about each of the steps to upload your files, check out [upload_nfts_steps](docs/upload_nfts_steps.md). See [multiple_asset_files](docs/multiple_asset_files.md) for an example of how to upload multiple media files.ok
+The single `upload-nfts` command below assumes you have a pair of image and metadata files for each of your NFTs. To learn more about each of the steps to upload your files, check out [upload_nfts_steps](docs/upload_nfts_steps.md). See [multiple_asset_files](docs/multiple_asset_files.md) for an example of how to upload multiple media files.
 
 ### Create Upload Folder
  Put your assets and associated metadata files with `.json` extension in a folder by themselves. You can use any kind of file you want. Arloader automatically adds a content type tag to your upload so that browsers will handle it correctly when accessed from Arweave.
@@ -179,7 +178,7 @@ Updating metadata manifest status...
 
 If you're uploading more than one file, you should pretty much always be using bundles. Bundles take multiple files and packages them together in a single transaction. This is better than uploading multiple individual files because you only have to wait for one transaction to be confirmed. Once the bundle transaction is confirmed, all of your files will be available. Larger transactions with larger rewards are more attractive to miners, which means a larger bundled transaction is more likely to get written quickly than a bunch of smaller individual ones.
 
-Arloader will create as many bundles as necessary to upload all of your files. Your files are read asynchronously, bundled in parallel across multiple threads and then posted to [arweave.net](https://arweave.net). Arloader supports bundle sizes up to 200 MB, but the default bundle size is 10 MB, which makes it possible to post full bundle size payloads to the `/tx` endpoint instead of in 256 KB chunks to the `/chunk` endpoint. This should work fine for individual files up to 10 MB. If your files sizes are bigger than 10 MB (but smaller than 200 MB), you can specify a larger bundle size with the `--bundles-size` argument - `--bundle-size 100` to specify a size of 100 MB, for example.
+Arloader will create as many bundles as necessary to upload all of your files. Your files are read asynchronously, bundled in parallel across multiple threads and then posted to [arweave.net](https://arweave.net). Arloader supports bundle sizes up to 200 MB, with a default of 10 MB, which makes it possible to post full bundle size payloads to the `/tx` endpoint instead of in 256 KB chunks to the `/chunk` endpoint. This should work fine for individual files up to 100 MB. If your files sizes are bigger than 100 MB (but smaller than 200 MB), you can specify a larger bundle size with the `--bundles-size` argument - `--bundle-size 200` to specify a size of 200 MB, for example. If you file sizes are bigger than 200 MB, you can upload them as individual files by passing the `--no-bundle` flag.
 
 ### Estimate Cost
 To get an estimate of the cost of uploading your files run
@@ -197,7 +196,7 @@ To upload your files run
 arloader upload <FILE_PATHS>
 ```
 
-This kicks off the process of uploading a stream of bundles created from your files. The default bundle size is 10 MB. The example output below had a bundle size of 5000 bytes.
+This kicks off the process of uploading a stream of bundles created from your files. The example output below had a bundle size of 5000 bytes.
 
 ```
 bundle txid                                   items      KB  status       confirms
@@ -430,7 +429,4 @@ file size | num files | arweave | bundlr | arweave total | bundlr total | arweav
 - [ ] Include duration in completion output.
 
 ## Potential Issue with Transactions Uploaded Prior to Version 0.1.51
-The way arloader was formatting transactions for upload was not entirely compatible with the Arweave protocol prior to version 1.51. For transactions bigger than 256 KB it is possible that even though your transactions are visible and are showing more than 25 confirmations that they were not written to the Arweave blockchain. If you would like assistance determining whether your transactions were impacted, please open an issue and I will be happy to help, including paying for any necessary re-uploading.
-
-## Tokens of Appreciation
-Tokens of appreciation can be sent to `F4B7659xdVcTqQEHShLsxp7w8wckMyBmT9GM8bGDqTUW`.
+The way arloader was formatting transactions for upload was not entirely compatible with the Arweave protocol prior to version 1.51. For transactions bigger than 256 KB it is possible that even though your transactions are visible and are showing more than 25 confirmations that they were not written to the Arweave blockchain. If you would like assistance determining whether your transactions were impacted, please open an issue and I will be happy to help, including paying for any necessary re-uploading.

examples/upload_files.rs

@@ -0,0 +1,95 @@
+use arloader::{commands::*, error::Error, status::OutputFormat, Arweave};
+use rayon::prelude::*;
+use std::env;
+use std::{fs, path::PathBuf, str::FromStr, time::Instant};
+use tempdir::TempDir;
+use url::Url;
+
+// For smaller sample sizes, you may have to increase this to have the transactions mined.
+const REWARD_MULTIPLIER: f32 = 2.0;
+const NUM_FILES: usize = 1;
+const FILE_SIZE: usize = 100_000_000;
+const BUFFER: usize = 5;
+
+#[tokio::main]
+async fn main() -> CommandResult {
+    let ar_keypair_path = env::var("AR_KEYPAIR_PATH").ok().map(PathBuf::from);
+    let sol_keypair_path = env::var("SOL_KEYPAIR_PATH").ok().map(PathBuf::from);
+
+    let arweave = if let Some(ar_keypair_path) = ar_keypair_path {
+        Arweave::from_keypair_path(
+            ar_keypair_path,
+            Url::from_str("https://arweave.net").unwrap(),
+        )
+        .await?
+    } else {
+        if sol_keypair_path.is_none() {
+            println!("Example requires either AR_KEYPAIR_PATH or SOL_KEYPAIR_PATH environment variable to be set.");
+            return Ok(());
+        };
+        Arweave::default()
+    };
+
+    let ext = "bin";
+    println!("Setting up files...\n");
+    let temp_dir = files_setup(FILE_SIZE, NUM_FILES, ext)?;
+    let paths_iter = (0..NUM_FILES).map(|i| temp_dir.path().join(format!("{}.bin", i)));
+    // let path_chunks = arweave.chunk_file_paths(paths_iter, BUNDLE_SIZE)?;
+    let log_dir = temp_dir.path().join("status/");
+    fs::create_dir(log_dir.clone()).unwrap();
+    let output_format = &OutputFormat::Display;
+
+    let start = Instant::now();
+    if sol_keypair_path.is_none() {
+        println!("Starting upload with AR...\n");
+        command_upload(
+            &arweave,
+            paths_iter,
+            Some(log_dir.clone()),
+            None,
+            REWARD_MULTIPLIER,
+            output_format,
+            BUFFER,
+        )
+        .await?;
+    } else {
+        println!("Starting upload with SOL...\n");
+        command_upload_with_sol(
+            &arweave,
+            paths_iter,
+            Some(log_dir.clone()),
+            None,
+            REWARD_MULTIPLIER,
+            output_format,
+            BUFFER,
+            sol_keypair_path.unwrap(),
+        )
+        .await?;
+    }
+
+    let duration = start.elapsed();
+
+    println!(
+        "\n\nUpload completed in: {:?}\n\nUpdating statuses..\n\n",
+        duration
+    );
+
+    let paths_iter = (0..NUM_FILES).map(|i| temp_dir.path().join(format!("{}.bin", i)));
+    command_update_statuses(&arweave, paths_iter, log_dir, output_format, 10).await?;
+    Ok(())
+}
+
+fn files_setup(file_size: usize, num_files: usize, ext: &str) -> Result<TempDir, Error> {
+    let bytes = vec![0; file_size];
+
+    let temp_dir = TempDir::new("tmp")?;
+
+    let _ = (0..num_files).into_par_iter().for_each(|i| {
+        fs::write(
+            temp_dir.path().join(format!("{}", i)).with_extension(ext),
+            &bytes,
+        )
+        .unwrap();
+    });
+    Ok(temp_dir)
+}

src/commands.rs

@@ -32,6 +32,7 @@ pub async fn command_files(paths: Option<Vec<PathBuf>>) -> CommandResult {
     println!("{:?}", paths);
     Ok(())
 }
+
 /// Gets cost of uploading a list of files.
 pub async fn command_get_cost<IP>(
     arweave: &Arweave,
@@ -1039,4 +1040,4 @@ pub fn get_manifest_id_from_log_dir(log_dir: &PathBuf) -> String {
         .nth(1)
         .unwrap()
         .to_string()
-}
+}

src/lib.rs

@@ -1042,7 +1042,12 @@ impl Arweave {
             )
             .await?;
         let signed_transaction = self.sign_transaction(transaction)?;
-        let (id, reward) = self.post_transaction(&signed_transaction).await?;
+        let (id, reward) = if signed_transaction.data.0.len() > MAX_TX_DATA as usize {
+            self.post_transaction_chunks(signed_transaction, 100)
+                .await?
+        } else {
+            self.post_transaction(&signed_transaction).await?
+        };
 
         let status = Status {
             id,
@@ -1099,7 +1104,12 @@ impl Arweave {
             .sign_transaction_with_sol(transaction, solana_url, sol_ar_url, from_keypair)
             .await?;
 
-        let (id, reward) = self.post_transaction(&signed_transaction).await?;
+        let (id, reward) = if signed_transaction.data.0.len() > MAX_TX_DATA as usize {
+            self.post_transaction_chunks(signed_transaction, 100)
+                .await?
+        } else {
+            self.post_transaction(&signed_transaction).await?
+        };
 
         let mut status = Status {
             file_path: Some(file_path),

src/main.rs

@@ -682,7 +682,7 @@ fn bundle_size_arg<'a, 'b>() -> Arg<'a, 'b> {
         .value_name("BUNDLE_SIZE")
         .takes_value(true)
         .validator(is_valid_bundle_size)
-        .default_value("10")
+        .default_value("100")
         .help("Specify the bundle size in megabytes.")
 }
 
@@ -955,6 +955,27 @@ fn is_json_file_path(path_str: String) -> Result<(), String> {
     }
 }
 
+// fn file_sizes_less_than_bundle_size(paths_vec: Vec<PathBuf>, bundle_size: u64) -> bool {
+//     if paths_vec.iter().any(|p| {
+//         let p_len = p.metadata().unwrap().len();
+//         if p_len > bundle_size {
+//             println!(
+//                 "{} is {} MB, which is greater than the bundle size of {}. Bundle size must be greater than file.",
+//                 p.display(),
+//                 p_len / 1_000_000,
+//                 bundle_size
+//             );
+//             true
+//         } else {
+//             false
+//         }
+//     }) {
+//         false
+//     } else {
+//         true
+//     }
+// }
+
 // ====================
 // Helpers
 // ====================