Service Lake Formation GetResourceLFTags Missing Table and Column Tags

[brascher]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

When calling the GetResourceLFTags() service in Lake Formation for a specific column, the LFTagsOnTable and LFTagsOnColumns slices in the GetResourceLFTagsOutput struct are empty.

Expected Behavior

All LF resource tags that are inherited from the database and table, along with any resource tags added to the column, are included in the GetResourceLFTagsOutput struct.

Current Behavior

LF resource tags inherited from the table and added to the column specifically are missing from the GetResourceLFTagsOutput struct.

Reproduction Steps

import (
	"context"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/lakeformation"
	"github.com/aws/aws-sdk-go-v2/service/lakeformation/types"
)

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithSharedConfigProfile("dev_profile"))

	svc := lakeformation.NewFromConfig(cfg)

	db := &types.DatabaseResource{
		Name:      aws.String("db_name"),
		CatalogId: aws.String("1234567890"),
	}
	tbl := &types.TableResource{
		DatabaseName: aws.String("db_name"),
		CatalogId:    aws.String("1234567890"),
		Name:         aws.String("tbl_name"),
	}
	cols := &types.TableWithColumnsResource{
		DatabaseName: aws.String("db_name"),
		CatalogId:    aws.String("1234567890"),
		Name:         aws.String("tbl_name"),
		ColumnNames:  []string {"distributionchannelcode"},
	}

	resource := &types.Resource{
		Database:         db,
		Table:            tbl,
		TableWithColumns: cols,
	}
	input := &lakeformation.GetResourceLFTagsInput{
		Resource:           resource,
		CatalogId:          aws.String("1234567890"),
		ShowAssignedLFTags: aws.Bool(true),
	}

	resp, err := svc.GetResourceLFTags(context.TODO(), input)
	if err != nil {
		log.Fatalf("Failed to list LF Tags, %v", err)
	}

	fmt.Printf("%+v", resp)

	fmt.Println("DB LF Tags")
	fmt.Println(len(resp.LFTagOnDatabase))
	for _, tt := range resp.LFTagOnDatabase {
		fmt.Printf("%s : %s in catalog %s \n", *tt.TagKey, tt.TagValues, *tt.CatalogId)
	}

	fmt.Println("Table LF Tags")
	fmt.Println(len(resp.LFTagsOnTable))
	for _, aa := range resp.LFTagsOnTable {
		fmt.Printf("%s : %s in catalog %s \n", *aa.TagKey, aa.TagValues, *aa.CatalogId)
	}

	fmt.Println("Column LF Tags")
	fmt.Println(len(resp.LFTagsOnColumns))
	for _, cc := range resp.LFTagsOnColumns {
		fmt.Printf("Column Name: %s \n", *cc.Name)
		fmt.Println(len(cc.LFTags))
		for _, jj := range cc.LFTags {
			fmt.Printf("%s : %s in catalog %s \n", *jj.TagKey, jj.TagValues, *jj.CatalogId)
		}
	}
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.30.3
github.com/aws/aws-sdk-go-v2/config v1.27.27
github.com/aws/aws-sdk-go-v2/service/lakeformation v1.35.3
github.com/aws/aws-sdk-go-v2/credentials v1.17.27 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.4 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 // indirect
github.com/aws/smithy-go v1.20.3 // indirect

Compiler and Version used

go version go1.19.8 windows/amd64

Operating System and version

Windows 10 Enterprise 22H2

[RanVaknin]

Hi @brascher,

Expected Behavior

All LF resource tags that are inherited from the database and table, along with any resource tags added to the column, are included in the GetResourceLFTagsOutput struct.

I was not able to find any documentation supporting this expected behavior. Is this something that you can see using the CLI?

My gut says this is not an SDK issue, but rather a service side behavior that is unclear.
If you enable the debug logs and look at raw response sent by the server (prior to deserialization) do you see the missing tags?

	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"), config.WithClientLogMode(aws.LogRequestWithBody | aws.LogResponseWithBody))

If when looking at the raw response you do see those tags, that would point to an issue in deserialization with SDK. If you do not see those tags, it would point to server side behavior that is either a bug , or expected behavior that is not well documented.

Thank you,
Ran~

[brascher]

Hi @RanVaknin ,

I should have posted this initially as I had gathered all this info at that time. Sorry about that. As you can see, both the AWS CLI and Python boto3 calls return the expected lists for LFTagsOnTable and LFTagsOnColumns, while the Go SDK calls returns empty lists for both.

Here is the AWS CLI output:

{
    "LFTagOnDatabase": [
        {
            "CatalogId": "1234567890",
            "TagKey": "owner",
            "TagValues": [
                "clinical"
            ]
        },
        {
            "CatalogId": "1234567890",
            "TagKey": "classification",
            "TagValues": [
                "phi"
            ]
        }
    ],
    "LFTagsOnTable": [
        {
            "CatalogId": "1234567890",
            "TagKey": "owner",
            "TagValues": [
                "clinical"
            ]
        },
        {
            "CatalogId": "1234567890",
            "TagKey": "classification",
            "TagValues": [
                "phi"
            ]
        }
    ],
    "LFTagsOnColumns": [
        {
            "Name": “col”_name,
            "LFTags": [
                {
                    "CatalogId": "1234567890",
                    "TagKey": "owner",
                    "TagValues": [
                        "clinical"
                    ]
                },
                {
                    "CatalogId": "1234567890",
                    "TagKey": "classification",
                    "TagValues": [
                        "Restricted"
                    ]
                }
            ]
        }
    ]
}

Here is the Python boto3 output:

{'ResponseMetadata': {'RequestId': 'e0499d6e-c42f-4c71-8436-c76b76538337', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Mon, 19 Aug 2024 15:10:30 GMT', 'content-type': 'application/json', 'content-length': '533', 'connection': 'keep-alive', 'x-amzn-requestid': 'e0499d6e-c42f-4c71-8436-c76b76538337', 'cache-control': 'no-cache'}, 'RetryAttempts': 0}, 'LFTagOnDatabase': [{'CatalogId': '1234567890', 'TagKey': 'owner', 'TagValues': ['clinical']}, {'CatalogId': '1234567890', 'TagKey': 'classification', 'TagValues': ['phi']}], 'LFTagsOnTable': [{'CatalogId': '1234567890', 'TagKey': 'owner', 'TagValues': ['clinical']}, {'CatalogId': '1234567890', 'TagKey': 'classification', 'TagValues': ['phi']}], 'LFTagsOnColumns': [{'Name': ‘col’_name, 'LFTags': [{'CatalogId': '1234567890', 'TagKey': 'owner', 'TagValues': ['clinical']}, {'CatalogId': '1234567890', 'TagKey': 'classification', 'TagValues': ['Restricted']}]}]}

Here is the aws-sdk-go-v2 output:

&{LFTagOnDatabase:[{TagKey:0xc0001c22b0 TagValues:[clinical] CatalogId:0xc0001c22a0 noSmithyDocumentSerde:{}} {TagKey:0xc0001c22d0 TagValues:[phi] CatalogId:0xc0001c22f0 noSmithyDocumentSerde:{}}] LFTagsOnColumns:[] LFTagsOnTable:[] ResultMetadata:{values:map[{}:-551571600 {}:0xc00000a1c0 {}:4413d688-6f37-40a6-a0cf-8cf481470364 {}:{wall:13954684498955227280 ext:532189401 loc:0x10151e0} {}:{wall:0 ext:63859686903 loc:<nil>} {}:{Results:[{Err:<nil> Retryable:false Retried:false ResponseMetadata:{values:map[{}:-551571600 {}:0xc00000a1c0 {}:4413d688-6f37-40a6-a0cf-8cf481470364 {}:{wall:13954684498955227280 ext:532189401 loc:0x10151e0} {}:{wall:0 ext:63859686903 loc:<nil>}]}}]}]} noSmithyDocumentSerde:{}}

[RanVaknin]

Hi @brascher ,

Thanks for the additional details.
It would be very helpful if you can share the request and response logs as I showed in my last comment.

Additionally since it seems to be working with the CLI can you do the same for the CLI? run that same command with the --debug flag and share the output with us.

Thanks,
Ran~

[brascher]

Hi @RanVaknin

Here is the CLI request:

2024-08-22 14:54:10,413 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetResourceLFTags) with params: {'url_path': '/GetResourceLFTags', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/json', 'User-Agent': 'aws-cli/2.17.22 md/awscrt#0.20.11 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#lakeformation.get-resource-lf-tags'}, 'body': b'{"CatalogId": "1234567890", "Resource": {"TableWithColumns": {"CatalogId": "1234567890", "DatabaseName": "db_name", "Name": "tbl_name", "ColumnNames": ["col_name"]}}}', 'url': '[https://lakeformation.us-east-1.amazonaws.com/GetResourceLFTags'](https://lakeformation.us-east-1.amazonaws.com/GetResourceLFTags%27), 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x0000019817444B10>, 'has_streaming_input': False, 'auth_type': None}}
2024-08-22 14:54:10,413 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/GetResourceLFTags

Here is the CLI response:

2024-08-22 14:54:11,780 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"LFTagOnDatabase":[{"CatalogId":"1234567890","TagKey":"owner","TagValues":["clinical"]},{"CatalogId":"1234567890","TagKey":"classification","TagValues":["phi"]}],"LFTagsOnColumns":[{"LFTags":[{"CatalogId":"1234567890","TagKey":"owner","TagValues":["clinical"]},{"CatalogId":"1234567890","TagKey":"classification","TagValues":["phi"]}],"Name":"col_name"}],"LFTagsOnTable":[{"CatalogId":"1234567890","TagKey":"owner","TagValues":["clinical"]},{"CatalogId":"1234567890","TagKey":"classification","TagValues":["phi"]}]}'

Here is the Go API request:

SDK 2024/08/22 15:21:25 DEBUG Request
POST /GetResourceLFTags HTTP/1.1
{"CatalogId":"1234567890","Resource":{"TableWithColumns":{"CatalogId":"1234567890","ColumnNames":["col_name"],"DatabaseName":"db_name","Name":"tbl_name"}},"ShowAssignedLFTags":true}

Here is the Go API response:

SDK 2024/08/22 15:21:26 DEBUG Response
HTTP/2.0 200 OK
Content-Length: 211
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 22 Aug 2024 19:21:27 GMT
X-Amzn-Requestid: 038a9d5d-7070-470c-a795-ea97bafebe46

{"LFTagOnDatabase":[{"CatalogId":"1234567890","TagKey":"owner","TagValues":["clinical"]},{"CatalogId":"1234567890","TagKey":"classification","TagValues":["phi"]}],"LFTagsOnColumns":null,"LFTagsOnTable":null}

Please let me know if you need anything else. Thanks.

[RanVaknin]

Hi @brascher ,

Thanks for getting back to me with those logs.

As you can see in the SDK logs the raw data coming back from the service contains LFTagsOnColumns":null,"LFTagsOnTable":null which explains why the SDK's deserialized response looks like LFTagsOnColumns:[] LFTagsOnTable:[].

You have redacted some metadata from the request,but from the information shared the main difference between the Go SDK and the CLi is that in Go you are sending ShowAssignedLFTags: true and in the CLI you don't send that value.

Doing my own reproduction I can see that indeed this flag would either return or not return the child tags:

with ShowAssignedLFTags:

{"LFTagOnDatabase":[{"CatalogId":"123456789012","TagKey":"classification","TagValues":["phi"]}],"LFTagsOnColumns":null,"LFTagsOnTable":[{"CatalogId":"123456789012","TagKey":"classification","TagValues":["phi"]}]}

without ShowAssignedLFTags:

{"LFTagOnDatabase":[{"CatalogId":"123456789012","TagKey":"classification","TagValues":["phi"]}],"LFTagsOnColumns":[{"LFTags":[{"CatalogId":"123456789012","TagKey":"classification","TagValues":["phi"]}],"Name":"distributionchannelcode"}],"LFTagsOnTable":[{"CatalogId":"123456789012","TagKey":"classification","TagValues":["phi"]}]}

Thanks,
Ran~