mount.txt

Mount
~~~~~

Before any other interaction can take place between a client and a
Lustre file system the client must 'mount' the file system, and Lustre
services must already be started on the servers. A file system
mount may be initiated via the 'mount()' system call, passing the
name of the MGS and Lustre file system to the kernel.  The Lustre
client exchanges a series of messages with the servers, beginning with
messages that retrieve the file system configuration <<llog>> from the
management server (MGS). This provides the client with the identities
of all the metadata servers (MDSs) and targets (MDTs) as well as all
the object storage servers (OSSs) and targets (OSTs). The client then
sequences through each of the targets exchanging additional messages
to initiate the connections with them. The following sections present
the details of the Lustre operations that accomplish the file system
mount.

Messages Between the Client and the MGS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to be able to mount the Lustre file system the client needs
to know the identities of the various servers and targets so that it
can initiate connections to them. The following sequence of operations
accomplishes this.

.Connect RPCs Between the Client and the MGS
[[client-mgs-connect-rpcs]
image::client_mgs_connect_rpcs.png["Connect RPCs Between the Client and the MGS", height=600]

//////////////////////////////////////////////////////////////////////
The client_mgs_connect_rpcs.png diagram resembles this text art:

Time
Step Client                         MGS                            MDT   OST
     -------                        -------                        ---   ---
1    MGS_CONNECT-------------------->
2              <--------------------MGS_CONNECT
3    LDLM_ENQUEUE------------------->
4               <-------------------LDLM_ENQUEUE
5    LLOG_ORIGIN_HANDLE_CREATE------>
6                            <------LLOG_ORIGIN_HANDLE_CREATE
7    LDLM_ENQUEUE------------------->
8               <-------------------LDLM_ENQUEUE
9    LLOG_ORIGIN_HANDLE_CREATE------>
10                           <------LLOG_ORIGIN_HANDLE_CREATE
11   LLOG_ORIGIN_HANDLE_READ_HEADER->
12                                <-LLOG_ORIGIN_HANDLE_READ_HEADER
13   LLOG_ORIGIN_HANDLE_NEXT_BLOCK->
14                               <-LLOG_ORIGIN_HANDLE_NEXT_BLOCK
15   LDLM_ENQUEUE------------------->
16              <-------------------LDLM_ENQUEUE
17   MGS_CONFIG_READ---------------->
18                 <----------------MGS_CONFIG_READ
19   LDLM_ENQUEUE------------------->
20              <-------------------LDLM_ENQUEUE
21   LLOG_ORIGIN_HANDLE_CREATE------>
22                           <------LLOG_ORIGIN_HANDLE_CREATE
23   LLOG_ORIGIN_HANDLE_READ_HEADER->
24                                <-LLOG_ORIGIN_HANDLE_READ_HEADER
//////////////////////////////////////////////////////////////////////

Prior to any other interaction between a client and a Lustre server
(or between two servers) the client must establish a 'connection'. The
connection establishes shared state between the two hosts. On the
client this connection state information is called an 'import', and
there is an import on the client for each target it connects to. On
the server this connection state is referred to as an 'export', and
again the server has an export for each client that has connected to
it.

*1 - Client issues an MGS_CONNECT to the MGS.*

.MGS_CONNECT Request Packet Structure
image::mgs-connect-request.png["MGS_CONNECT Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mgs-connect-request.png diagram resembles this text art:

        MGS_CONNECT:
      --request--------------------------------------------------
      | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
      -----------------------------------------------------------
      |  obd_connect_data |
      ---------------------
//////////////////////////////////////////////////////////////////////

See <<mgs-connect-rpc>>.

The client begins by carrying out the <<mgs-connect-rpc,MGS_CONNECT>>
Lustre RPC, which establishes the connection (creates the import and
the export) between the client and the MGS.  The connect message from
the client includes a 'lustre_handle' to uniquely identify itself
(note that this is distinct from the 'pb_handle' field of the
structure described in <<struct-ptlrpc-body>>, which will be 0 in a
connection request). The target notes the 'lustre_handle' value in its
'export'. The connection data from the client also proposes the set of
<<connect-flags,connection flags>> appropriate to connecting to an
MGS.

.Flags for the client connection to an MGS
[options="header"]
|====
| obd_connect_data->ocd_connect_flags
| OBD_CONNECT_VERSION
| OBD_CONNECT_AT
| OBD_CONNECT_FULL20
| OBD_CONNECT_IMP_RECOV
| OBD_CONNECT_MNE_SWAB
| OBD_CONNECT_PINGLESS
|====

*2 - The MGS sends an MGS_CONNECT reply to the client.*

.MGS_CONNECT Reply Packet Structure
image::mgs-connect-reply.png["MGS_CONNECT Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mgs-connect-reply.png diagram resembles this text art:

        MGS_CONNECT:
      --reply---------------------------
      | ptlrpc_body | obd_connect_data |
      ----------------------------------
//////////////////////////////////////////////////////////////////////

The MGS's reply to the connection request will include the
'lustre_handle' (in the 'ptlrpc_body' structure's 'pb_handle' field)
that the client will use (in its 'pb_handle') to identify this target
in subsequent messages. The MGS also replies with the same set of
connection flags.

Once the connection is established the client gets configuration
information for the file system from the MGS in four stages. First,
the two exchange messages establishing the file system wide security
policy that will be followed in all subsequent communications.

Each time the client is going to read information from server storage
it needs to first acquire the appropriate lock. Since the client is
only reading data, the locks will be 'concurrent read' locks. The
LDLM_ENQUEUE command communicates this lock request to the MGS
target. The request identifies the target via the connection-handle
from the connection reply, and identifies the client (itself) with the
client-handle from its original connection request.  The MGS's reply
grants that lock, if appropriate. If other clients were making some
sort of modification to the MGS data then the lock exchange might
result in a delay while the client waits. More details about the
behavior of the <<ldlm,Distributed Lock Manager>> are in that
section.

*3 - The Client sends an LDLM_ENQUEUE to the MGS.*

.LDLM_ENQUEUE Request Packet Structure
image::ldlm-enqueue-request.png["LDLM_ENQUEUE Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The ldlm-enqueue-request.png diagram resembles this text art:

       LDLM_ENQUEUE:
      --request---------------------
      | ptlrpc_body | ldlm_request |
      ------------------------------
//////////////////////////////////////////////////////////////////////

See <<ldlm-enqueue-rpc>>.

The client seeks a 'concurrent read' lock on the MGS.

*4 - The MGS sends an LDLM_ENQUEUE reply to the client.*

.LDLM_ENQUEUE Reply Packet Structure
image::ldlm-enqueue-reply.png["LDLM_ENQUEUE Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The ldlm-enqueue-reply.png diagram resembles this text
art:

       LDLM_ENQUEUE:
      --reply---------------------
      | ptlrpc_body | ldlm_reply |
      ----------------------------
//////////////////////////////////////////////////////////////////////

The MGS grants that lock.

*5 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*

.LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure
image::llog-origin-handle-create-request.png["LLOG_ORIGIN_HANDLE_CREATE Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-create-request.png diagram resembles this text
art:

       LLOG_ORIGIN_HANDLE_CREATE:
      --request----------------------------
      | ptlrpc_body | llogd_body | string |
      -------------------------------------
//////////////////////////////////////////////////////////////////////

See <<llog-origin-handle-create>>.

The first LLOG_ORIGIN_HANDLE_CREATE operation asks
for the security configuration file ("lfs-sptlrpc"). <<security>>
discusses security.

*6 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*


.LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure
image::llog-origin-handle-create-reply.png["LLOG_ORIGIN_HANDLE_CREATE Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-create-reply.png diagram resembles this text
art:

       LLOG_ORIGIN_HANDLE_CREATE:
      --reply---------------------
      | ptlrpc_body | llogd_body |
      ----------------------------
//////////////////////////////////////////////////////////////////////

In this case there is nothing to be done for security. That is,
subsequent messages will all use an "empty security flavor" and no
encryption will take place. In this case the MGS's reply ('pb_status'
== -2, ENOENT) indicated that there was no such file, so nothing
actually gets read.

Next, the client gets a configuration log (See <<llog>>) starting with
a bitmap instructing it as to which among the configuration records on
the MGS it needs. Another LDLM_ENQUEUE and LLOG_ORIGIN_HANDLE_CREATE
pair of operations identifies the configuration client data
("lfs-client") file.

*7 - The Client sends an LDLM_ENQUEUE to the MGS.*

See <<ldlm-enqueue-rpc>>.

The client again seeks a 'concurrent read' lock on the MGS.

*8 - The MGS sends an LDLM_ENQUEUE reply to the client.*

The MGS grants that lock.

*9 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*

See <<llog-origin-handle-create>>.

The client asks for the 'lfs-client' log file, which holds a bitmap
indicating the available configuration records.

*10 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*

In this case there is data to read. The LLOG_ORIGIN_HANDLE_CREATE
reply identifies the actual object of interest on the MGS via the
'llog_logid' field in the 'struct llogd_body'.

*11 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*

.LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure
image::llog-origin-handle-read-header-request.png["LLOG_ORIGIN_HANDLE_READ_HEADER Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-read-header-request.png diagram resembles this
text art:

       LLOG_ORIGIN_HANDLE_READ_HEADER:
      --request-------------------
      | ptlrpc_body | llogd_body |
      ----------------------------
//////////////////////////////////////////////////////////////////////

See <<llog-origin-handle-read-header>>.

The client asks for that log file's header. The MGS stores
configuration data in log records. A header at the beginning of
"lfs-client" uses a bitmap to identify the log records that are
actually needed.  The header includes both which records to retrieve
and how large those records are. The LLOG_ORIGIN_HANDLE_READ_HEADER
request uses the 'llog_logid' to identify desired log file, and the
reply provides the bitmap and size information identifying the
records that are actually needed.

*12 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*

.LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure
image::llog-origin-handle-read-header-reply.png["LLOG_ORIGIN_HANDLE_READ_HEADER Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-read-header-reply.png diagram resembles this
text art:

       LLOG_ORIGIN_HANDLE_READ_HEADER:
      --reply--------------------------
      | ptlrpc_body | llog_log_header |
      ---------------------------------
//////////////////////////////////////////////////////////////////////

The MGs responds with the header, which holds the actual bitmap.

*13 - The Client sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK to the MGS.*

.LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure
image::llog-origin-handle-next-block-request.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-next-block-request.png diagram resembles this
text art:

       LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
      --request-------------------
      | ptlrpc_body | llogd_body |
      ----------------------------
//////////////////////////////////////////////////////////////////////

See <<llog-origin-handle-next-block>>.

The client asks for the configuration record.

*14 - The MGS sends an LLOG_ORIGIN_HANDLE_NEXT_BLOCK reply to the client.*

.LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure
image::llog-origin-handle-next-block-reply.png["LLOG_ORIGIN_HANDLE_NEXT_BLOCK Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The llog-origin-handle-next-block-reply.png diagram resembles this
text art:

       LLOG_ORIGIN_HANDLE_NEXT_BLOCK:
      --reply------------------------------
      | ptlrpc_body | llogd_body | eadata |
      -------------------------------------
//////////////////////////////////////////////////////////////////////

The MGS replies with the details of where in the configuration data to
find the desired details.

Knowing the specific configuration records it wants, the client then
proceeds to retrieve them. This requires another LDLM_ENQUEUE
operation, followed this time by the MGS_CONFIG_READ operation, which
get the UUIDs for the servers and targets from the configuration log
("lfs-cliir").

*15 - The Client sends an LDLM_ENQUEUE to the MGS.*

See <<ldlm-enqueue-rpc>>.

The client again seeks a 'concurrent read' lock on the MGS.

*16 - The MGS sends an LDLM_ENQUEUE reply to the client.*

The MGS grants that lock.

*17 - The Client sends an MGS_CONFIG_READ to the MGS.*

.MGS_CONFIG_READ Request Packet Structure
image::mgs-config-read-request.png["MGS_CONFIG_READ Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mgs-config-read-request.png diagram resembles this text art:

        MGS_CONFIG_READ:
      --request------------------------
      | ptlrpc_body | mgs_config_body |
      ---------------------------------
//////////////////////////////////////////////////////////////////////

See <<mgs-config-read-rpc>>.

The client identifies the desired record in the 'lfs-cliir' file,
which contains the current details of the configuration for this file
system.

*18 - The MGS sends an MGS_CONFIG_READ reply to the client.*

.MGS_CONFIG_READ Reply Packet Structure
image::mgs-config-read-reply.png["MGS_CONFIG_READ Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mgs-config-read-reply.png diagram resembles this text art:

        MGS_CONFIG_READ:
      --reply-------------------------
      | ptlrpc_body | mgs_config_res |
      --------------------------------
//////////////////////////////////////////////////////////////////////

The MGS responds with the actual configuration data. This gives the
client the list of all the servers and targets it will need to
communicate with.

Finally, the client reads the cluster wide configuration data (the
sort that might be set at the client command line with a 'lctl
conf_param' command). The following paragraphs go into these four
stages in more detail.

*19 - The Client sends an LDLM_ENQUEUE to the MGS.*

See <<ldlm-enqueue-rpc>>.

The client again seeks a 'concurrent read' lock on the MGS.

*20 - The MGS sends an LDLM_ENQUEUE reply to the client.*

The MGS grants that lock.

*21 - The Client sends an LLOG_ORIGIN_HANDLE_CREATE to the MGS.*

See <<llog-origin-handle-create>>.

The client asks for the 'params' log file.

*22 - The MGS sends an LLOG_ORIGIN_HANDLE_CREATE reply to the client.*

The MGS responds that the log file is available.

*23 - The Client sends an LLOG_ORIGIN_HANDLE_READ_HEADER to the MGS.*

See <<llog-origin-handle-read-header>>.

The client asks for that log file's header.

*24 - The MGS sends an LLOG_ORIGIN_HANDLE_READ_HEADER reply to the client.*

The MGs responds with the header, which holds the actual bitmap. In
this case there are no 'params' to report.

Messages Between the Client and the MDSs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

After the foregoing interaction with the MGS the client has a list of
the MDSs and MDTs in the file system.  Next, the client invokes four
Lustre operations with each MDT on the list.

.Connect RPCs Between the Client and each MDT
[[client-mdt-connect-rpcs]
image::client_mdt_connect_rpcs.png["Connect RPCs Between the Client and each MDT", height=200]

//////////////////////////////////////////////////////////////////////
The client_mdt_connect_rpcs.png diagram resembles this text art:

Time
Step Client         MGS             MDT            OST
     -------        -------         --------       ------
1    MDS_CONNECT-------------------->
2              <--------------------MDS_CONNECT
3    MDS_STATFS--------------------->
4             <---------------------MDS_STATFS
5    MDS_GETSTATUS------------------>
6                <------------------MDS_GETSTATUS
7    MDS_GETATTR-------------------->
8              <--------------------MDS_GETATTR
//////////////////////////////////////////////////////////////////////

The MDS_CONNECT operation establishes a connection between the client
and a specific target (MDT) on an MDS. Thus, if an MDS has multiple
targets, there is a separate MDS_CONNECT operation for each. This
creates an import for the target on the client and an export for the
client and target on the MDS.

*1 - Client issues an MDS_CONNECT to each MDT.*

.MDS_CONNECT Request Packet Structure
image::mds-connect-request.png["MDS_CONNECT Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-connect-request.png diagram resembles this text art:

        MDS_CONNECT:
      --request--------------------------------------------------
      | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
      -----------------------------------------------------------
      |  obd_connect_data |
      ---------------------
//////////////////////////////////////////////////////////////////////

See <<mds-connect-rpc>>.

As with the connect operation for the MGS, the connect message from
the client includes a 'lustre_handle' to uniquely identify this
connection. The connection data from the client also proposes the set
of <<connect-flags,connection flags>> appropriate to connecting to an
MDS. The following are the flags always included.

.Always included flags for the client connection to an MDS
[options="header"]
|====
| obd_connect_data->ocd_connect_flags
| OBD_CONNECT_RDONLY
| OBD_CONNECT_VERSION
| OBD_CONNECT_ACL
| OBD_CONNECT_XATTR
| OBD_CONNECT_IBITS
| OBD_CONNECT_NODEVOH
| OBD_CONNECT_ATTRFID
| OBD_CONNECT_CANCELSET
| OBD_CONNECT_AT
| OBD_CONNECT_RMT_CLIENT
| OBD_CONNECT_RMT_CLIENT_FORCE
| OBD_CONNECT_BRW_SIZE
| OBD_CONNECT_MDS_CAPA
| OBD_CONNECT_OSS_CAPA
| OBD_CONNECT_MDS_MDS
| OBD_CONNECT_FID
| LRU_RESIZE_CONNECT_FLAG
| OBD_CONNECT_VBR
| OBD_CONNECT_LOV_V3
| OBD_CONNECT_SOM
| OBD_CONNECT_FULL20
| OBD_CONNECT_64BITHASH
| OBD_CONNECT_JOBSTATS
| OBD_CONNECT_EINPROGRESS
| OBD_CONNECT_LIGHTWEIGHT
| OBD_CONNECT_UMASK
| OBD_CONNECT_LVB_TYPE
| OBD_CONNECT_LAYOUTLOCK
| OBD_CONNECT_PINGLESS
| OBD_CONNECT_MAX_EASIZE
| OBD_CONNECT_FLOCK_DEAD
| OBD_CONNECT_DISP_STRIPE
| OBD_CONNECT_LFSCK
| OBD_CONNECT_OPEN_BY_FID
| OBD_CONNECT_DIR_STRIPE
|====

.Optional flags for the client connection to an MDS
[options="header"]
|====
| obd_connect_data->ocd_connect_flags
| OBD_CONNECT_SOM
| OBD_CONNECT_LRU_RESIZE
| OBD_CONNECT_ACL
| OBD_CONNECT_UMASK
| OBD_CONNECT_RDONLY
| OBD_CONNECT_XATTR
| OBD_CONNECT_XATTR
| OBD_CONNECT_RMT_CLIENT_FORCE
|====

*2 - The MDS sends an MDS_CONNECT reply to the client.*

.MDS_CONNECT Reply Packet Structure
image::mds-connect-reply.png["MDS_CONNECT Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-connect-reply.png diagram resembles this text art:

        MDS_CONNECT:
      --reply---------------------------
      | ptlrpc_body | obd_connect_data |
      ----------------------------------
//////////////////////////////////////////////////////////////////////

The MDS replies to the connect message with a subset of the flags
proposed by the client, and the client notes those values in its
import.  As with the MGS, the MDS's reply to the connection request
will include a 'lustre_handle' in the 'pb_handle' field that the
client will use to identify this target in subsequent messages.

*3 - The Client sends an MDS_STATFS to the MDS.*

.MDS_STATFS Request Packet Structure
image::mds-statfs-request.png["MDS_STATFS Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-statfs-request.png diagram resembles this text art:

       MDS_STATFS:
      --request------
      | ptlrpc_body |
      ---------------
//////////////////////////////////////////////////////////////////////

See <<mds-statfs-rpc>>.

The client next uses an MDS_STATFS operation to request 'statfs'
information from the target, and that data is returned in the reply
message. The actual fields closely resemble the results of a 'statfs'
system call. See the 'obd_statfs' structure in the <<data-structs,Data
Structures and Defines Section>>.

*4 - The MDS sends an MDS_STATFS reply to the client.*

.MDS_STATFS Reply Packet Structure
image::mds-statfs-reply.png["MDS_STATFS Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-statfs-reply.png diagram resembles this text art:

       MDS_STATFS:
      --reply---------------------
      | ptlrpc_body | obd_statfs |
      ----------------------------
//////////////////////////////////////////////////////////////////////

The MDS replies with the 'statfs' information.

*5 - The Client sends an MDS_GETSTATUS to the MDS.*

.MDS_GETSTATUS Request Packet Structure
image::mds-getstatus-request.png["MDS_GETSTATUS Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-getstatus-request.png diagram resembles this text art:

       MDS_GETSTATUS:
      --request-----------------
      | ptlrpc_body | mdt_body |
      --------------------------
//////////////////////////////////////////////////////////////////////

See <<mds-getstatus-rpc>>.

The client uses the MDS_GETSTATUS operation to request information
about the mount point of the file system.

*6 - The MDS sends an MDS_GETSTATUS reply to the client.*

.MDS_GETSTATUS Reply Packet Structure
image::mds-getstatus-reply.png["MDS_GETSTATUS Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-getstatus-reply.png diagram resembles this text art:

       MDS_GETSTATUS:
      --reply-------------------
      | ptlrpc_body | mdt_body |
      --------------------------
//////////////////////////////////////////////////////////////////////

The server reply contains the 'fid' of the root directory of the file
system being mounted. If there is a security policy the capabilities
of that security policy are included in the reply.

*7 - The Client sends an MDS_GETATTR to the MDS.*

.MDS_GETATTR Request Packet Structure
image::mds-getattr-request.png["MDS_GETATTR Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-getattr-request.png diagram resembles this text art:

       MDS_GETATTR:
      --request-------------------------------
      | ptlrpc_body | mdt_body | lustre_capa |
      ----------------------------------------
//////////////////////////////////////////////////////////////////////

See <<mds-getattr-rpc>>.

The client then uses the MDS_GETATTR operation to get get further
information about the root directory of the file system. The request
message includes the above fid.

*8 - The MDS sends an MDS_GETATTR reply to the client.*

.MDS_GETATTR Reply Packet Structure
image::mds-getattr-reply.png["MDS_GETATTR Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The mds-getattr-reply.png diagram resembles this text art:

       MDS_GETATTR:
      --reply------------------------------------------------
      | ptlrpc_body | mdt_body | MDS_MD | ACL | lustre_capa |
      -------------------------------------------------------
      | lustre_capa |
      ---------------
//////////////////////////////////////////////////////////////////////

It will also include the security capability (if appropriate). The
reply also holds the same fid, and in this case the 'mdt_body' has
several additional fields filled in. These include the mtime, atime,
ctime, mode, uid, and gid. It also includes the size of the extended
attributes and the size of the ACL information. The reply message also
includes the extended attributes and the ACL. From the extended
attributes the client can find out about striping information for the
root, if any.

Messages Between the Client and the OSSs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Additional CONNECT messages flow between the client and each OST
enumerated by the MGS.

.Connect RPCs Between the Client and each OST
[[client-ost-connect-rpcs]
image::client_ost_connect_rpcs.png["Connect RPCs Between the Client and each OST", height=75]

//////////////////////////////////////////////////////////////////////
The client_ost_connect_rpcs.png diagram resembles this text art:

Time
Step Client         MGS             MDT            OST
     -------        -------         --------       ------
1    OST_CONNECT----------------------------------->
2              <-----------------------------------OST_CONNECT
//////////////////////////////////////////////////////////////////////

The OST_CONNECT operation establishes a connection between the client
and a specific target (OST) on an OST. Thus, if an OST has multiple
targets, there is a separate OST_CONNECT operation for each. This
creates an import for the target on the client and an export for the
client and target on the OST.

*1 - Client issues an OST_CONNECT to each OST.*

.OST_CONNECT Request Packet Structure
image::ost-connect-request.png["OST_CONNECT Request Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The ost-connect-request.png diagram resembles this text art:

        OST_CONNECT:
      --request--------------------------------------------------
      | ptlrpc_body | target_uuid | client_uuid | lustre_handle |
      -----------------------------------------------------------
      |  obd_connect_data |
      ---------------------
//////////////////////////////////////////////////////////////////////

See <<ost-connect-rpc>>.

As with the connect operations for the MGS and MDTs, the connect
message from the client includes a 'lustre_handle' to uniquely
identify this connection. The connection data from the client also
proposes the set of <<connect-flags,connection flags>> appropriate to
connecting to an OST. The following are the flags always included.

.Flags for the client connection to an OST
[options="header"]
|====
| obd_connect_data->ocd_connect_flags
| OBD_CONNECT_GRANT
| OBD_CONNECT_SRVLOCK
| OBD_CONNECT_VERSION
| OBD_CONNECT_REQPORTAL
| OBD_CONNECT_TRUNCLOCK
| OBD_CONNECT_RMT_CLIENT
| OBD_CONNECT_BRW_SIZE
| OBD_CONNECT_OSS_CAPA
| OBD_CONNECT_CANCELSET
| OBD_CONNECT_AT
| OBD_CONNECT_LRU_RESIZE
| OBD_CONNECT_CKSUM
| OBD_CONNECT_FID
| OBD_CONNECT_VBR
| OBD_CONNECT_FULL20
| OBD_CONNECT_LAYOUTLOCK
| OBD_CONNECT_64BITHASH
| OBD_CONNECT_MAXBYTES
| OBD_CONNECT_JOBSTATS
| OBD_CONNECT_EINPROGRESS
| OBD_CONNECT_LVB_TYPE
| OBD_CONNECT_PINGLESS
|====

*2 - The OST sends an OST_CONNECT reply to the client.*

.OST_CONNECT Reply Packet Structure
image::ost-connect-reply.png["OST_CONNECT Reply Packet Structure",height=50]

//////////////////////////////////////////////////////////////////////
The ost-connect-reply.png diagram resembles this text art:

        OST_CONNECT:
      --reply---------------------------
      | ptlrpc_body | obd_connect_data |
      ----------------------------------
//////////////////////////////////////////////////////////////////////

The OST replies to the connect message with a subset of the flags
proposed by the client, and the client notes those values in its
import.  The OST's reply to the connection request will include a UUID
that the server and client will both use to identify this connection
in subsequent messages. The flags that the server doesn't include are:
OBD_CONNECT_RMT_CLIENT, OBD_CONNECT_OSS_CAPA, and
OBD_CONNECT_PINGLESS.