Skip to content

Commit 95eef94

Browse files
ritalrwritalrw
committed
Update README.
1 parent 83fb6ba commit 95eef94

1 file changed

Lines changed: 158 additions & 47 deletions

File tree

‎README.md‎

Lines changed: 158 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,160 @@
11
# DocLoader
22

3-
Run:
4-
```
5-
mvn compile exec:java -Dexec.cleanupDaemonThreads=false -Dexec.mainClass="couchbase.test.sdk.Loader" -Dexec.args="-n <ip> -user <username> -pwd <password> -b <bucket-name> -p 11210 -create_s 0 -create_e 10000000 -cr 100 -ops 100000 -docSize 1024 -scope <> -collection <>"
6-
```
7-
8-
```
9-
Required options: n, user, pwd, b, p
10-
usage: Supported Options
11-
-b,--bucket <arg> Bucket
12-
-collection <arg> Collection
13-
-cr,--create <arg> Creates%
14-
-create_e,--create_e <arg> Creates Start
15-
-create_s,--create_s <arg> Creates Start
16-
-delete_e,--delete_e <arg> Delete End
17-
-delete_s,--delete_s <arg> Delete Start
18-
-deleted,--deleted <arg> To verify deleted docs
19-
-dl,--delete <arg> Deletes%
20-
-docSize,--docSize <arg> Size of the doc
21-
-durability <arg> Durability Level
22-
-ex,--expiry <arg> Expiry%
23-
-expiry_e,--expiry_e <arg> Expiry End
24-
-expiry_s,--expiry_s <arg> Expiry Start
25-
-gtm,--gtm <arg> Go for max doc ops
26-
-keySize,--keySize <arg> Size of the key
27-
-keyType,--keyType <arg> Random/Sequential/Reverse
28-
-loadType,--loadType <arg> Hot/Cold
29-
-n,--node <arg> IP Address
30-
-ops,--ops <arg> Ops/Sec
31-
-p,--port <arg> Memcached Port
32-
-pwd,--rest_password <arg> Password
33-
-rd,--read <arg> Reads%
34-
-read_e,--read_e <arg> Read End
35-
-read_s,--read_s <arg> Read Start
36-
-replace_e,--replace_w <arg> Replace End
37-
-replace_s,--replace_s <arg> Replace Start
38-
-scope <arg> Scope
39-
-touch_e,--touch_e <arg> Touch End
40-
-touch_s,--touch_s <arg> Touch Start
41-
-transaction_patterns <arg> Transaction load pattern
42-
-up,--update <arg> Updates%
43-
-update_e,--update_e <arg> Update End
44-
-update_s,--update_s <arg> Update Start
45-
-user,--rest_username <arg> Username
46-
-validate,--validate <arg> Validate Data during Reads
47-
-valueType,--valueType <arg>
48-
-w,--workers <arg> Workers
49-
```
3+
## Overview
4+
5+
DocLoader is a comprehensive Java-based document loading and testing tool designed
6+
for Couchbase, MongoDB, and Elasticsearch databases. It provides flexible workload
7+
generation capabilities for performance testing, data loading, and validation
8+
scenarios across multiple NoSQL databases.
9+
10+
## Features
11+
12+
• Multi-database support (Couchbase, MongoDB, Elasticsearch)
13+
• Configurable workload patterns (Create, Read, Update, Delete, Expiry)
14+
• Subdocument operations support
15+
• Transaction testing capabilities
16+
• Customizable document and key generation
17+
• Batch processing with configurable batch sizes
18+
• Performance monitoring and metrics
19+
• Support for various data types and document structures
20+
21+
## Prerequisites
22+
23+
• Java 8 or higher
24+
• Maven 3.6+
25+
• Access to target database (Couchbase/MongoDB/Elasticsearch)
26+
• Network connectivity to database servers
27+
28+
## Build Instructions
29+
30+
1. Clone the repository
31+
2. Navigate to the project directory
32+
3. Build the project:
33+
```bash
34+
mvn clean compile
35+
```
36+
37+
## Running the Application
38+
39+
### Couchbase Loader:
40+
```bash
41+
mvn compile exec:java -Dexec.cleanupDaemonThreads=false -Dexec.mainClass="Loader" -Dexec.args="-n <ip> -user <username> -pwd <password> -b <bucket-name> -p 11210 -create_s 0 -create_e 10000000 -cr 100 -ops 100000 -docSize 1024 -scope <> -collection <>"
42+
```
43+
44+
### MongoDB Loader:
45+
```bash
46+
mvn compile exec:java -Dexec.mainClass="MongoLoader" -Dexec.args="<connection-string> <database> <collection>"
47+
```
48+
49+
### SIFT Loader:
50+
```bash
51+
mvn compile exec:java -Dexec.mainClass="SIFTLoader" -Dexec.args="<parameters>"
52+
```
53+
54+
## Command Line Options
55+
56+
### Required Options:
57+
- `-n, --node <arg>` IP Address of the database server
58+
- `-user, --rest_username <arg>` Username for authentication
59+
- `-pwd, --rest_password <arg>` Password for authentication
60+
- `-b, --bucket <arg>` Bucket name (Couchbase)
61+
- `-p, --port <arg>` Memcached Port
62+
63+
### Optional Options:
64+
- `-cr, --create <arg>` Percentage of create operations
65+
- `-create_s, --create_s <arg>` Create operations start range
66+
- `-create_e, --create_e <arg>` Create operations end range
67+
- `-rd, --read <arg>` Percentage of read operations
68+
- `-read_s, --read_s <arg>` Read operations start range
69+
- `-read_e, --read_e <arg>` Read operations end range
70+
- `-up, --update <arg>` Percentage of update operations
71+
- `-update_s, --update_s <arg>` Update operations start range
72+
- `-update_e, --update_e <arg>` Update operations end range
73+
- `-dl, --delete <arg>` Percentage of delete operations
74+
- `-delete_s, --delete_s <arg>` Delete operations start range
75+
- `-delete_e, --delete_e <arg>` Delete operations end range
76+
- `-ex, --expiry <arg>` Percentage of expiry operations
77+
- `-expiry_s, --expiry_s <arg>` Expiry operations start range
78+
- `-expiry_e, --expiry_e <arg>` Expiry operations end range
79+
- `-docSize, --docSize <arg>` Size of documents in bytes
80+
- `-keySize, --keySize <arg>` Size of keys in bytes
81+
- `-keyType, --keyType <arg>` Key generation type (Random/Sequential/Reverse)
82+
- `-valueType, --valueType <arg>` Value generation type
83+
- `-ops, --ops <arg>` Operations per second
84+
- `-w, --workers <arg>` Number of worker threads
85+
- `-batchSize, --batchSize <arg>` Batch size for operations
86+
- `-scope <arg>` Scope name (Couchbase)
87+
- `-collection <arg>` Collection name (Couchbase)
88+
- `-durability <arg>` Durability level
89+
- `-validate, --validate <arg>` Validate data during reads
90+
- `-loadType, --loadType <arg>` Load type (Hot/Cold)
91+
- `-gtm, --gtm <arg>` Go for max document operations
92+
- `-transaction_patterns <arg>` Transaction load patterns
93+
94+
## Project Structure
95+
96+
```
97+
src/main/java/
98+
├── Loader.java # Main Couchbase loader
99+
├── MongoLoader.java # MongoDB loader
100+
├── SIFTLoader.java # SIFT-specific loader
101+
├── couchbase/ # Couchbase-specific utilities
102+
├── mongo/ # MongoDB-specific utilities
103+
├── elasticsearch/ # Elasticsearch-specific utilities
104+
├── RestServer/ # REST server components
105+
└── utils/
106+
├── docgen/ # Document generation utilities
107+
│ └── DocumentGenerator.java # Document generation logic
108+
├── key/ # Key generation utilities
109+
└── val/ # Value generation utilities
110+
```
111+
112+
## Key Components
113+
114+
• **DocumentGenerator**: Handles document creation and management
115+
• **Key Generators**: RandomKey, SimpleKey, CircularKey, ReverseKey
116+
• **Value Generators**: Various document templates (Hotel, Cars, Product, etc.)
117+
• **Workload Settings**: Configuration management for test parameters
118+
• **Database Connectors**: Couchbase, MongoDB, Elasticsearch clients
119+
120+
## Usage Examples
121+
122+
### Basic Couchbase Load Test:
123+
```bash
124+
mvn compile exec:java -Dexec.mainClass="couchbase.test.sdk.Loader" -Dexec.args="-n localhost -user Administrator -pwd password -b test-bucket -p 11210 -create_s 0 -create_e 10000 -cr 100 -ops 1000 -docSize 1024"
125+
```
126+
127+
### Read-Heavy Workload:
128+
```bash
129+
mvn compile exec:java -Dexec.mainClass="couchbase.test.sdk.Loader" -Dexec.args="-n localhost -user Administrator -pwd password -b test-bucket -p 11210 -read_s 0 -read_e 10000 -rd 80 -up 20 -ops 5000"
130+
```
131+
132+
### Mixed Workload with Updates:
133+
```bash
134+
mvn compile exec:java -Dexec.mainClass="couchbase.test.sdk.Loader" -Dexec.args="-n localhost -user Administrator -pwd password -b test-bucket -p 11210 -create_s 0 -create_e 5000 -cr 30 -rd 40 -up 30 -ops 2000"
135+
```
136+
137+
## Troubleshooting
138+
139+
• Ensure database server is running and accessible
140+
• Verify network connectivity and firewall settings
141+
• Check authentication credentials
142+
• Monitor system resources (CPU, memory, network)
143+
• Review logs for detailed error information
144+
145+
## Performance Tips
146+
147+
• Start with smaller batch sizes and gradually increase
148+
• Monitor system resources during load testing
149+
• Use appropriate worker thread counts based on system capacity
150+
• Consider network latency when testing remote databases
151+
• Use appropriate document sizes for your use case
152+
153+
## License
154+
155+
This project is part of the Couchbase testing suite.
156+
157+
## Support
158+
159+
For issues and questions, please refer to the Couchbase documentation
160+
or contact the development team.

0 commit comments

Comments
 (0)