Skip to content

[Storage] [STG 98] Merge STG 98 Base Branch into Main #40381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
94d112b
[Storage] [STG 98] Branch Prep (#39392)
weirongw23-msft Jan 28, 2025
a47b5d6
[Storage] [STG 98] OAuth Tests for Access Control Policies APIs (#39446)
weirongw23-msft Feb 1, 2025
ce93425
[Storage] [STG 98] NFS over REST 98 (#39438)
weirongw23-msft Feb 1, 2025
ef8a89d
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Feb 11, 2025
2937934
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Feb 12, 2025
6b0e358
[Storage] [STG 98] Added support for `x-ms-file-request-intent` for B…
weirongw23-msft Feb 13, 2025
c0d6ba1
Replaced %2F in recordings with / (#39660)
weirongw23-msft Feb 13, 2025
c5a867f
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Feb 22, 2025
a4d10cd
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Mar 18, 2025
1083425
[Storage] [STG 98] API Views Feedback (#40123)
weirongw23-msft Mar 20, 2025
02bd20d
Added progress_hook to datalake upload_data, download_file (#40170)
weirongw23-msft Mar 21, 2025
4583682
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Apr 1, 2025
fc643c8
[Storage] [Named Keywords] [Queue] Transition to Named Keywords for `…
weirongw23-msft Apr 1, 2025
143e43d
[Storage] [STG 98] Beta Changelogs (#40312)
weirongw23-msft Apr 3, 2025
98e1f05
Regenerated swagger for 98 beta (#40364)
weirongw23-msft Apr 4, 2025
5a9e52c
Async query blob changelogs
weirongw23-msft Apr 4, 2025
d4e9f20
Async query blob (#40378)
weirongw23-msft Apr 4, 2025
070590f
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Apr 5, 2025
1ea27e9
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Apr 5, 2025
ca7f1a0
Merge branch 'main' into feature/storage-stg98
weirongw23-msft Apr 7, 2025
173207c
[Storage] [STG 98] Refactored File -> Blob Copy Tests to Use `request…
weirongw23-msft Apr 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions sdk/storage/azure-storage-blob/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
## 12.26.0b1 (Unreleased)

### Features Added
- Added support for service version 2025-07-05.
- Added support for OAuth authentication in `ContainerClient`'s `get_container_access_policy`
and `set_container_access_policy` APIs.
- Added support for the keyword `source_token_intent` when copying from file share to blob and
authenticated via `TokenCredential` for the following `BlobClient` APIs: `upload_blob_from_url`,
`start_copy_from_url`, `stage_block_from_url`, `upload_pages_from_url`, and `append_block_from_url`.
- Added support for `query_blob` in the asynchronous `BlobClient`.

## 12.25.1 (2025-03-27)

Expand Down
2 changes: 1 addition & 1 deletion sdk/storage/azure-storage-blob/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "python",
"TagPrefix": "python/storage/azure-storage-blob",
"Tag": "python/storage/azure-storage-blob_b11831f46e"
"Tag": "python/storage/azure-storage-blob_7756dd1c10"
}
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,15 @@ def upload_blob_from_url(
:keyword str source_authorization:
Authenticate as a service principal using a client secret to access a source blob. Ensure "bearer " is
the prefix of the source_authorization string.
:keyword source_token_intent:
Required when source is Azure Storage Files and using `TokenCredential` for authentication.
This is ignored for other forms of authentication.
Specifies the intent for all requests when using `TokenCredential` authentication. Possible values are:

backup - Specifies requests are intended for backup/admin type operations, meaning that all file/directory
ACLs are bypassed and full permissions are granted. User must also have required RBAC permission.

:paramtype source_token_intent: Literal['backup']
:returns: Blob-updated property Dict (Etag and last modified)
:rtype: Dict[str, Any]
"""
Expand All @@ -430,7 +439,8 @@ def upload_blob_from_url(
options = _upload_blob_from_url_options(
source_url=source_url,
metadata=metadata,
**kwargs)
**kwargs
)
try:
return cast(Dict[str, Any], self._client.block_blob.put_blob_from_url(**options))
except HttpResponseError as error:
Expand Down Expand Up @@ -1746,6 +1756,15 @@ def start_copy_from_url(

.. versionadded:: 12.9.0

:keyword source_token_intent:
Required when source is Azure Storage Files and using `TokenCredential` for authentication.
This is ignored for other forms of authentication.
Specifies the intent for all requests when using `TokenCredential` authentication. Possible values are:

backup - Specifies requests are intended for backup/admin type operations, meaning that all file/directory
ACLs are bypassed and full permissions are granted. User must also have required RBAC permission.

:paramtype source_token_intent: Literal['backup']
:keyword str encryption_scope:
A predefined encryption scope used to encrypt the data on the sync copied blob. An encryption
scope can be created using the Management API and referenced here by name. If a default
Expand All @@ -1770,7 +1789,8 @@ def start_copy_from_url(
source_url=source_url,
metadata=metadata,
incremental_copy=incremental_copy,
**kwargs)
**kwargs
)
try:
if incremental_copy:
return cast(Dict[str, Union[str, datetime]], self._client.page_blob.copy_incremental(**options))
Expand Down Expand Up @@ -2046,6 +2066,15 @@ def stage_block_from_url(
:keyword str source_authorization:
Authenticate as a service principal using a client secret to access a source blob. Ensure "bearer " is
the prefix of the source_authorization string.
:keyword source_token_intent:
Required when source is Azure Storage Files and using `TokenCredential` for authentication.
This is ignored for other forms of authentication.
Specifies the intent for all requests when using `TokenCredential` authentication. Possible values are:

backup - Specifies requests are intended for backup/admin type operations, meaning that all file/directory
ACLs are bypassed and full permissions are granted. User must also have required RBAC permission.

:paramtype source_token_intent: Literal['backup']
:returns: Blob property dict.
:rtype: dict[str, Any]
"""
Expand All @@ -2057,7 +2086,8 @@ def stage_block_from_url(
source_offset=source_offset,
source_length=source_length,
source_content_md5=source_content_md5,
**kwargs)
**kwargs
)
try:
return cast(Dict[str, Any], self._client.block_blob.stage_block_from_url(**options))
except HttpResponseError as error:
Expand Down Expand Up @@ -2919,6 +2949,15 @@ def upload_pages_from_url(
:keyword str source_authorization:
Authenticate as a service principal using a client secret to access a source blob. Ensure "bearer " is
the prefix of the source_authorization string.
:keyword source_token_intent:
Required when source is Azure Storage Files and using `TokenCredential` for authentication.
This is ignored for other forms of authentication.
Specifies the intent for all requests when using `TokenCredential` authentication. Possible values are:

backup - Specifies requests are intended for backup/admin type operations, meaning that all file/directory
ACLs are bypassed and full permissions are granted. User must also have required RBAC permission.

:paramtype source_token_intent: Literal['backup']
:returns: Response after uploading pages from specified URL.
:rtype: Dict[str, Any]
"""
Expand Down Expand Up @@ -3211,6 +3250,15 @@ def append_block_from_url(
:keyword str source_authorization:
Authenticate as a service principal using a client secret to access a source blob. Ensure "bearer " is
the prefix of the source_authorization string.
:keyword source_token_intent:
Required when source is Azure Storage Files and using `TokenCredential` for authentication.
This is ignored for other forms of authentication.
Specifies the intent for all requests when using `TokenCredential` authentication. Possible values are:

backup - Specifies requests are intended for backup/admin type operations, meaning that all file/directory
ACLs are bypassed and full permissions are granted. User must also have required RBAC permission.

:paramtype source_token_intent: Literal['backup']
:returns: Result after appending a new block.
:rtype: Dict[str, Union[str, datetime, int]]
"""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ def _upload_blob_from_url_options(source_url: str, **kwargs: Any) -> Dict[str, A
overwrite = kwargs.pop('overwrite', False)
content_settings = kwargs.pop('content_settings', None)
source_authorization = kwargs.pop('source_authorization', None)
source_token_intent = kwargs.pop('source_token_intent', None)
if content_settings:
kwargs['blob_http_headers'] = BlobHTTPHeaders(
blob_cache_control=content_settings.cache_control,
Expand All @@ -214,6 +215,7 @@ def _upload_blob_from_url_options(source_url: str, **kwargs: Any) -> Dict[str, A

options = {
'copy_source_authorization': source_authorization,
'file_request_intent': source_token_intent,
'content_length': 0,
'copy_source_blob_properties': kwargs.pop('include_source_blob_properties', True),
'source_content_md5': kwargs.pop('source_content_md5', None),
Expand Down Expand Up @@ -376,7 +378,7 @@ def _quick_query_options(snapshot: Optional[str], query_expression: str, **kwarg
'timeout': kwargs.pop('timeout', None),
'cls': return_headers_and_deserialized,
}
options.update(kwargs)
options.update({k: v for k, v in kwargs.items() if v is not None})
return options, delimiter

def _generic_delete_blob_options(delete_snapshots: Optional[str] = None, **kwargs: Any) -> Dict[str, Any]:
Expand Down Expand Up @@ -607,6 +609,7 @@ def _start_copy_from_url_options( # pylint:disable=too-many-statements
requires_sync = kwargs.pop('requires_sync', None)
encryption_scope_str = kwargs.pop('encryption_scope', None)
source_authorization = kwargs.pop('source_authorization', None)
source_token_intent = kwargs.pop('source_token_intent', None)
# If tags is a str, interpret that as copy_source_tags
copy_source_tags = isinstance(tags, str)

Expand All @@ -626,6 +629,8 @@ def _start_copy_from_url_options( # pylint:disable=too-many-statements
headers['x-ms-encryption-scope'] = encryption_scope_str
if source_authorization:
headers['x-ms-copy-source-authorization'] = source_authorization
if source_token_intent:
headers['x-ms-file-request-intent'] = source_token_intent
if copy_source_tags:
headers['x-ms-copy-source-tag-option'] = tags
else:
Expand All @@ -635,6 +640,9 @@ def _start_copy_from_url_options( # pylint:disable=too-many-statements
if source_authorization:
raise ValueError(
"Source authorization tokens are only supported for sync copy, please specify requires_sync=True")
if source_token_intent:
raise ValueError(
"Source token intent is only supported for sync copy, please specify requires_sync=True")
if copy_source_tags:
raise ValueError(
"Copying source tags is only supported for sync copy, please specify requires_sync=True")
Expand Down Expand Up @@ -729,6 +737,7 @@ def _stage_block_from_url_options(
) -> Dict[str, Any]:
source_url = _encode_source_url(source_url=source_url)
source_authorization = kwargs.pop('source_authorization', None)
source_token_intent = kwargs.pop('source_token_intent', None)
if source_length is not None and source_offset is None:
raise ValueError("Source offset value must not be None if length is set.")
if source_length is not None and source_offset is not None:
Expand All @@ -747,6 +756,7 @@ def _stage_block_from_url_options(
encryption_algorithm=cpk.algorithm)
options = {
'copy_source_authorization': source_authorization,
'file_request_intent': source_token_intent,
'block_id': block_id,
'content_length': 0,
'source_url': source_url,
Expand Down Expand Up @@ -1010,6 +1020,7 @@ def _upload_pages_from_url_options(
if_sequence_number_equal_to=kwargs.pop('if_sequence_number_eq', None)
)
source_authorization = kwargs.pop('source_authorization', None)
source_token_intent = kwargs.pop('source_token_intent', None)
access_conditions = get_access_conditions(kwargs.pop('lease', None))
mod_conditions = get_modify_conditions(kwargs)
source_mod_conditions = get_source_conditions(kwargs)
Expand All @@ -1023,6 +1034,7 @@ def _upload_pages_from_url_options(

options = {
'copy_source_authorization': source_authorization,
'file_request_intent': source_token_intent,
'source_url': source_url,
'content_length': 0,
'source_range': source_range,
Expand Down Expand Up @@ -1152,6 +1164,7 @@ def _append_block_from_url_options(
append_position=appendpos_condition
)
source_authorization = kwargs.pop('source_authorization', None)
source_token_intent = kwargs.pop('source_token_intent', None)
access_conditions = get_access_conditions(kwargs.pop('lease', None))
mod_conditions = get_modify_conditions(kwargs)
source_mod_conditions = get_source_conditions(kwargs)
Expand All @@ -1164,6 +1177,7 @@ def _append_block_from_url_options(

options = {
'copy_source_authorization': source_authorization,
'file_request_intent': source_token_intent,
'source_url': copy_source_url,
'content_length': 0,
'source_range': source_range,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ class AzureBlobStorage: # pylint: disable=client-accepts-api-version-keyword
:param base_url: Service URL. Required. Default value is "".
:type base_url: str
:keyword version: Specifies the version of the operation to use for this request. Default value
is "2025-01-05". Note that overriding this default value may result in unsupported behavior.
is "2025-07-05". Note that overriding this default value may result in unsupported behavior.
:paramtype version: str
"""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,12 @@ class AzureBlobStorageConfiguration: # pylint: disable=too-many-instance-attrib
desired operation. Required.
:type url: str
:keyword version: Specifies the version of the operation to use for this request. Default value
is "2025-01-05". Note that overriding this default value may result in unsupported behavior.
is "2025-07-05". Note that overriding this default value may result in unsupported behavior.
:paramtype version: str
"""

def __init__(self, url: str, **kwargs: Any) -> None:
version: Literal["2025-01-05"] = kwargs.pop("version", "2025-01-05")
version: Literal["2025-07-05"] = kwargs.pop("version", "2025-07-05")

if url is None:
raise ValueError("Parameter 'url' must not be None.")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# pylint: disable=too-many-lines
# pylint: disable=line-too-long,useless-suppression,too-many-lines
# --------------------------------------------------------------------------
#
# Copyright (c) Microsoft Corporation. All rights reserved.
Expand Down Expand Up @@ -411,7 +411,7 @@ def from_dict(
:param function key_extractors: A key extractor function.
:param str content_type: JSON by default, set application/xml if XML.
:returns: An instance of this model
:raises: DeserializationError if something went wrong
:raises DeserializationError: if something went wrong
:rtype: Self
"""
deserializer = Deserializer(cls._infer_class_models())
Expand Down Expand Up @@ -1361,7 +1361,7 @@ def xml_key_extractor(attr, attr_desc, data): # pylint: disable=unused-argument
# Iter and wrapped, should have found one node only (the wrap one)
if len(children) != 1:
raise DeserializationError(
"Tried to deserialize an array not wrapped, and found several nodes '{}'. Maybe you should declare this array as wrapped?".format( # pylint: disable=line-too-long
"Tried to deserialize an array not wrapped, and found several nodes '{}'. Maybe you should declare this array as wrapped?".format(
xml_name
)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ class AzureBlobStorage: # pylint: disable=client-accepts-api-version-keyword
:param base_url: Service URL. Required. Default value is "".
:type base_url: str
:keyword version: Specifies the version of the operation to use for this request. Default value
is "2025-01-05". Note that overriding this default value may result in unsupported behavior.
is "2025-07-05". Note that overriding this default value may result in unsupported behavior.
:paramtype version: str
"""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,12 @@ class AzureBlobStorageConfiguration: # pylint: disable=too-many-instance-attrib
desired operation. Required.
:type url: str
:keyword version: Specifies the version of the operation to use for this request. Default value
is "2025-01-05". Note that overriding this default value may result in unsupported behavior.
is "2025-07-05". Note that overriding this default value may result in unsupported behavior.
:paramtype version: str
"""

def __init__(self, url: str, **kwargs: Any) -> None:
version: Literal["2025-01-05"] = kwargs.pop("version", "2025-01-05")
version: Literal["2025-07-05"] = kwargs.pop("version", "2025-07-05")

if url is None:
raise ValueError("Parameter 'url' must not be None.")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# pylint: disable=line-too-long,useless-suppression
# coding=utf-8
# --------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
Expand Down Expand Up @@ -78,7 +79,6 @@ async def create(
modified_access_conditions: Optional[_models.ModifiedAccessConditions] = None,
**kwargs: Any
) -> None:
# pylint: disable=line-too-long
"""The Create Append Blob operation creates a new append blob.

:param content_length: The length of the request. Required.
Expand Down Expand Up @@ -264,7 +264,6 @@ async def append_block(
modified_access_conditions: Optional[_models.ModifiedAccessConditions] = None,
**kwargs: Any
) -> None:
# pylint: disable=line-too-long
"""The Append Block operation commits a new block of data to the end of an existing append blob.
The Append Block operation is permitted only if the blob was created with x-ms-blob-type set to
AppendBlob. Append Block is supported only on version 2015-02-21 version or later.
Expand Down Expand Up @@ -445,6 +444,7 @@ async def append_block_from_url(
transactional_content_md5: Optional[bytes] = None,
request_id_parameter: Optional[str] = None,
copy_source_authorization: Optional[str] = None,
file_request_intent: Optional[Union[str, _models.FileShareTokenIntent]] = None,
cpk_info: Optional[_models.CpkInfo] = None,
cpk_scope_info: Optional[_models.CpkScopeInfo] = None,
lease_access_conditions: Optional[_models.LeaseAccessConditions] = None,
Expand All @@ -453,7 +453,6 @@ async def append_block_from_url(
source_modified_access_conditions: Optional[_models.SourceModifiedAccessConditions] = None,
**kwargs: Any
) -> None:
# pylint: disable=line-too-long
"""The Append Block operation commits a new block of data to the end of an existing append blob
where the contents are read from a source url. The Append Block operation is permitted only if
the blob was created with x-ms-blob-type set to AppendBlob. Append Block is supported only on
Expand Down Expand Up @@ -486,6 +485,8 @@ async def append_block_from_url(
:param copy_source_authorization: Only Bearer type is supported. Credentials should be a valid
OAuth access token to copy source. Default value is None.
:type copy_source_authorization: str
:param file_request_intent: Valid value is backup. "backup" Default value is None.
:type file_request_intent: str or ~azure.storage.blob.models.FileShareTokenIntent
:param cpk_info: Parameter group. Default value is None.
:type cpk_info: ~azure.storage.blob.models.CpkInfo
:param cpk_scope_info: Parameter group. Default value is None.
Expand Down Expand Up @@ -584,6 +585,7 @@ async def append_block_from_url(
source_if_none_match=_source_if_none_match,
request_id_parameter=request_id_parameter,
copy_source_authorization=copy_source_authorization,
file_request_intent=file_request_intent,
comp=comp,
version=self._config.version,
headers=_headers,
Expand Down Expand Up @@ -642,7 +644,6 @@ async def seal(
append_position_access_conditions: Optional[_models.AppendPositionAccessConditions] = None,
**kwargs: Any
) -> None:
# pylint: disable=line-too-long
"""The Seal operation seals the Append Blob to make it read-only. Seal is supported only on
version 2019-12-12 version or later.

Expand Down
Loading
Loading