Skip to content

Commit c853356

Browse files
committed
Improve http_host, filename in docs and example/http-s3.fio
In fio.1 and HOWTO.rst: * http_host: add details on vHost- vs Path-style hostname differences * filename: add details on bucket prefix for Path-style hostname In examples/http-s3.fio: * add comments at the top to clarify vHost vs Path S3 and change the hostname in existing example to disambiguate two scenarios Signed-off-by: Sean Lee <[email protected]>
1 parent 8ac0eec commit c853356

File tree

3 files changed

+34
-7
lines changed

3 files changed

+34
-7
lines changed

HOWTO.rst

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -805,6 +805,11 @@ Target file/device
805805
Note: Windows and FreeBSD (refer to geom(4)) prevent write access to areas
806806
of the disk containing in-use data (e.g. filesystems).
807807

808+
For HTTP and S3 access, specify a valid URL path or S3 key, respectively.
809+
A filename for path-style S3 includes a bucket name (:file:`/bucket/k/e.y`)
810+
while a virtual-hosted-style S3 filename :file:`/k/e.y` does not because
811+
its bucket name is specified in :option:`http_host`.
812+
808813
The filename "`-`" is a reserved name, meaning *stdin* or *stdout*. Which
809814
of the two depends on the read/write direction set.
810815

@@ -2934,8 +2939,13 @@ with the caveat that when used on the command line, they must come after the
29342939

29352940
.. option:: http_host=str : [http]
29362941

2937-
Hostname to connect to. For S3, this could be the bucket hostname.
2938-
Default is **localhost**
2942+
Hostname to connect to. HTTP port 80 is used automatically when the value of
2943+
the https parameter is *off*, and HTTPS port 443 if it is *on*. A
2944+
virtual-hosted-style S3 hostname starts with a bucket name, while a
2945+
path-style S3 hostname does not. See
2946+
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html for
2947+
detailed examples.
2948+
Default is **localhost** (path-style S3 hostname)
29392949

29402950
.. option:: http_user=str : [http]
29412951

examples/http-s3.fio

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,29 @@
11
# Example test for the HTTP engine's S3 support against Amazon AWS.
22
# Obviously, you have to adjust the S3 credentials; for this example,
33
# they're passed in via the environment.
4-
#
4+
# For non-AWS S3 implementations, refer to your S3 vendor's region
5+
# settings. Note that the region value appears twice, in http_host and
6+
# http_s3_region.
7+
# This example uses virtual-hosted-style requests:
8+
# https://bucket-name.s3.region-code.amazonaws.com/k/e.y
9+
# For path-style, prefix the key with the bucket name in the filename
10+
# so that filename=/bucket-name/k/e.y:
11+
# https://s3.region-code.amazonaws.com/bucket-name/k/e.y
12+
13+
# Reference for Virtual-hosted-style vs. Path-style URLs:
14+
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
515

616
[global]
717
ioengine=http
818
name=test
919
direct=1
10-
filename=/larsmb-fio-test/object
20+
filename=/k/e.y
1121
http_verbose=0
1222
https=on
1323
http_mode=s3
1424
http_s3_key=${S3_KEY}
1525
http_s3_keyid=${S3_ID}
16-
http_host=s3.eu-central-1.amazonaws.com
26+
http_host=bucket-name.s3.eu-central-1.amazonaws.com
1727
http_s3_region=eu-central-1
1828
group_reporting
1929

fio.1

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -585,6 +585,11 @@ the first device, `\\\\.\\PhysicalDrive1' for the second etc.
585585
Note: Windows and FreeBSD prevent write access to areas
586586
of the disk containing in-use data (e.g. filesystems).
587587
.P
588+
For HTTP and S3 access, specify a valid URL path or S3 key, respectively.
589+
A filename for path-style S3 includes a bucket name (`/bucket/k/e.y')
590+
while a virtual-hosted-style S3 filename (`/k/e.y') does not because its
591+
bucket name is specified in \fBhttp_host\fR.
592+
.P
588593
The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which
589594
of the two depends on the read/write direction set.
590595
.RE
@@ -2554,8 +2559,10 @@ Touching all objects affects ceph caches and likely impacts test results.
25542559
Enabled by default.
25552560
.TP
25562561
.BI (http)http_host \fR=\fPstr
2557-
Hostname to connect to. For S3, this could be the bucket name. Default
2558-
is \fBlocalhost\fR
2562+
Hostname to connect to. HTTP port 80 is used automatically when the value
2563+
of the \fBhttps\fP parameter is \fRoff\fP, and HTTPS port 443 if it is \Ron\fP.
2564+
A virtual-hosted-style S3 hostname starts with a bucket name, while a
2565+
path-style S3 hostname does not. Default is \fBlocalhost\fR.
25592566
.TP
25602567
.BI (http)http_user \fR=\fPstr
25612568
Username for HTTP authentication.

0 commit comments

Comments
 (0)