Added Private Endpoint deployment option #609

patkje75 · 2024-02-19T12:49:35Z

Added code for Private Endpoints and updated the .\finops-hub\README.md
Added parameters to storage.bicep

🛠️ Description

Possibility to do deploy FinOps hub with private endpoints.

🔬 How did you test this change?

🤏 Lint tests

🤞 PS -WhatIf / az validate

👍 Manually deployed + verified

💪 Unit tests

🙌 Integration tests

🙋‍♀️ Do any of the following that apply?

🚨 This is a breaking change.

🤏 The change is less than 20 lines of code.

📑 Did you update `docs/changelog.md`?

✅ Updated changelog (required for dev PRs)

➡️ Will add log in a future PR (feature branch PRs only)

❎ Log not needed (small/internal change)

📖 Did you update documentation?

✅ Public docs in docs (required for dev)

✅ Internal dev docs in src (required for dev)

➡️ Will add docs in a future PR (feature branch PRs only)

❎ Docs not needed (small/internal change)

patkje75 · 2024-02-19T14:52:31Z

@patkje75 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

@microsoft-github-policy-service agree

…param.

flanakin

Sorry for the delay! Too much going on 😓

src/templates/finops-hub/README.md

src/templates/finops-hub/modules/keyVault.bicep

src/templates/finops-hub/modules/storage.bicep

src/templates/finops-hub/modules/hub.bicep

sebassem

LGTM, however I have some comments in addition to Michael's

Use only the subnetResourceId to indicate a private deployment and drop the publicNetworkAccess parameter
If subnetResourceId is specified, lets assess if we can have the deployment scripts also deployed in private, see this and this . Update the readme accordingly if applicable. We will probably need to have a dedicated storage account for scripts since this scenario only supports service endpoints
Move the private endpoint deployment for keyVault into the keyVault module for better code organization

bradlovas · 2024-04-24T14:18:03Z

Hi team, any updates on this. Lots of customers extremely interested in deploying this however not being able to deploy with private endpoints is a blocker for them as it's against policy. Thanks!

patkje75 · 2024-04-25T21:48:50Z

Sorry, I had to down prioritize this due to a hectic schedule, I hope I can look into this and update the requested changes within the coming weeks.

…ed PVE to specific module

… be disabled for PaaS services policy

patkje75 · 2024-05-08T15:57:48Z

Sorry for the delay I have had a very busy time and I will soon be out for a while. I hope I have adressed all you comments.

fetching latest

…oyment

sebassem · 2024-05-15T16:42:30Z

Sorry for the delay I have had a very busy time and I will soon be out for a while. I hope I have adressed all you comments.

No worries, I just submitted a PR to your fork with some changes. Please review and test if possible so we can finalize and merge :)

CC: @flanakin @arthurclares

Added deployment scripts private access

…torage already has a private endpoint

patkje75 · 2024-05-22T15:31:55Z

@sebassem deploying the resources work, however the deployment script fails

It tries to access the FinOps storage account . I haven't had the time to look into all changes you have made, I only removed a Private Endpoint for blob storage because one already exsists.
I am sorry but I wont't have the time to dig into this, I am leaving for holidays in two weeks and I have to finish up two large projects, sorry.

sebassem · 2024-05-22T16:05:31Z

@sebassem deploying the resources work, however the deployment script fails

It tries to access the FinOps storage account . I haven't had the time to look into all changes you have made, I only removed a Private Endpoint for blob storage because one already exsists. I am sorry but I wont't have the time to dig into this, I am leaving for holidays in two weeks and I have to finish up two large projects, sorry.

I assume you don't have a private dns zone for blob storage ? Without it the deployment script won't be able to access it

patkje75 · 2024-05-22T20:09:37Z

You were right, it was the DNS, we have policies managing registration of private endpoints to the private DNS zones and it hadn't applied yet. I was trying to do to many things at once and didn't check that, sorry.

It seem to work now, I noticed the the name of the export directory had changed from msexports to only exports.
Probably thats why it didn't work the first time I did deploy, since that time th DNS registration worked and everything seemed OK but the pipeline wasn't triggered, now I know why.

I have updated the README.md as well with the new expected name of the export folder.

patkje75 · 2024-05-23T05:08:01Z

Too fast, I saw the pipeline kick off and left, today I see that i failed.

Operation on target Convert CSV failed: ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'AvailabilityZone' specified in column mapping cannot be found in 'part_0_0001.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'

Seems like the column AvailabilityZone is missing in the CSV-file. I am using the Amortized cost metric in the Export as written in the README.md.

sebassem · 2024-05-23T06:16:24Z

You were right, it was the DNS, we have policies managing registration of private endpoints to the private DNS zones and it hadn't applied yet. I was trying to do to many things at once and didn't check that, sorry.

It seem to work now, I noticed the the name of the export directory had changed from msexports to only exports. Probably thats why it didn't work the first time I did deploy, since that time th DNS registration worked and everything seemed OK but the pipeline wasn't triggered, now I know why.

I have updated the README.md as well with the new expected name of the export folder.

No worries, thanks for testing this. If the name change affects other parts of the code we can set it back to msexports, maybe I renamed it by mistake. Please let me know when you are happy with your testing so we can merge :)

sebassem · 2024-05-23T06:31:36Z

Too fast, I saw the pipeline kick off and left, today I see that i failed. Operation on target Convert CSV failed: ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'AvailabilityZone' specified in column mapping cannot be found in 'part_0_0001.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'

Seems like the column AvailabilityZone is missing in the CSV-file. I am using the Amortized cost metric in the Export as written in the README.md.

@flanakin @arthurclares

src/templates/finops-hub/README.md

Co-authored-by: Brett Wilson <[email protected]>

MSBrett · 2024-06-04T14:20:11Z

src/templates/finops-hub/README.md

+#### Deployment
+
+1. Login
+   > az login


Please change this to Powershell so it aligns with the module we're shipping.

MSBrett · 2024-06-04T14:20:56Z

src/templates/finops-hub/README.md

+2. Set target subscription
+   > az account set -s `<SubscriptionId>`
+3. Execute deployment
+   > az deployment group create --resource-group `<RG Name>` --location `<location>` --template-file .\main.bicep --parameters hubName='`<hubName>`' subnetResourceId='`<subnetResourceId>`' scriptsSubnetResourceId='`<scriptsSubnetResourceId>`'


Please change to Powershell

MSBrett · 2024-06-04T14:27:19Z

src/templates/finops-hub/README.md

+- The private deployment assumes that the target environment already has the virtual network where the private endpoints will be deployed.
+  - The virtual network needs to have two subnets; one for the private endpoints and one for the deployment scripts needed for the FinOps Hub.
+  - The virtual network for the deployment scripts needs to have the [needed configuration](https://learn.microsoft.com/azure/azure-resource-manager/bicep/deployment-script-vnet).
+    - A service endpoint configured for `Microsoft.Storage`
+    - A subnet delegation for `Microsoft.ContainerInstance/containerGroups`


Could provide this as part of the deployment so customers can retro-fit to their environment rather than leaving it undefined.

@MSBrett are you suggesting deploying the needed infra (vnet, dns zones ,....etc) ? And should it be deployed by default or with a flag ?

I'd prefer to just ship it at a vnet which can be peered to a hub as required. What's the minimum address space we need for this?

I would like to have the option to use an existing VNet as well, because deploying a new VNet for a single purpose won't fit all environments, however, this will require some more work, but will give an option for all.

fair enough

Patric Kjelldorff added 2 commits February 15, 2024 16:40

Added code for Private Endpoints and updated the .\finops-hub\README.md

4b087de

Added parameters to storage.bicep

b3b946f

patkje75 requested a review from MSBrett as a code owner February 19, 2024 12:49

microsoft-github-policy-service bot requested a review from flanakin February 19, 2024 12:49

microsoft-github-policy-service bot assigned flanakin Feb 19, 2024

microsoft-github-policy-service bot added the Tool: FinOps hubs Data pipeline solution label Feb 19, 2024

microsoft-github-policy-service bot assigned MSBrett Feb 19, 2024

microsoft-github-policy-service bot added the Needs: Review 👀 PR that is ready to be reviewed label Feb 19, 2024

flanakin requested review from Springstone, sebassem and arthurclares February 20, 2024 00:36

flanakin assigned Springstone, sebassem and arthurclares Feb 20, 2024

Removed VNet integration for storage account and Integration Runtime …

d82c19d

…param.

flanakin requested changes Mar 17, 2024

View reviewed changes

microsoft-github-policy-service bot added the Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity label Mar 17, 2024

microsoft-github-policy-service bot assigned patkje75 Mar 17, 2024

flanakin removed their assignment Mar 17, 2024

sebassem requested changes Mar 19, 2024

View reviewed changes

flanakin added this to the 0.3 milestone Mar 26, 2024

flanakin modified the milestones: 0.4, Hubs - Azure Landing Zones support Apr 3, 2024

Patric Kjelldorff added 4 commits May 8, 2024 17:13

Change so that the resourceId parameter disable public access and mov…

c055e32

…ed PVE to specific module

Added link to AzInitiativeAdvertizer for Public network access should…

9a388f8

… be disabled for PaaS services policy

Removed Disable Public Access parameter from main.bicep

6ea9416

Added Subnet Resource Id to UI Deployment template

9edd9f8

microsoft-github-policy-service bot requested a review from flanakin May 8, 2024 15:52

microsoft-github-policy-service bot assigned flanakin May 8, 2024

sebassem added 3 commits May 15, 2024 12:22

Merge pull request microsoft#7 from patkje75/features/private

6919860

fetching latest

chore: Update main.bicep and keyVault.bicep for Private Endpoint depl…

9bd7532

…oyment

update readme

dd2f3ee

patkje75 and others added 3 commits May 21, 2024 14:43

Merge pull request #1 from sebassem/sebassem/features/private

09f3e02

Added deployment scripts private access

Added location parameter to deployment string

e49fa09

Removed unecessary private endpoint from stroage accaont since blob s…

7fa3705

…torage already has a private endpoint

Changed the export directory in the README.md from msexports to exports

fe710db

flanakin modified the milestones: Hubs - Azure Landing Zones support, June 2024 May 23, 2024

MSBrett reviewed Jun 3, 2024

View reviewed changes

src/templates/finops-hub/README.md Outdated Show resolved Hide resolved

Update src/templates/finops-hub/README.md

254ff9e

Co-authored-by: Brett Wilson <[email protected]>

microsoft-github-policy-service bot requested a review from MSBrett June 4, 2024 06:50

MSBrett reviewed Jun 4, 2024

View reviewed changes

flanakin modified the milestones: July 2024, Hubs - Azure Landing Zones support Jun 24, 2024

microsoft-github-policy-service bot requested a review from MSBrett July 27, 2024 03:27

flanakin removed the Needs: Review 👀 PR that is ready to be reviewed label Aug 16, 2024

flanakin requested a review from sebassem September 13, 2024 18:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Private Endpoint deployment option #609

Added Private Endpoint deployment option #609

patkje75 commented Feb 19, 2024 •

edited by flanakin

Loading

patkje75 commented Feb 19, 2024

flanakin left a comment

sebassem left a comment •

edited

Loading

bradlovas commented Apr 24, 2024

patkje75 commented Apr 25, 2024

patkje75 commented May 8, 2024

sebassem commented May 15, 2024

patkje75 commented May 22, 2024

sebassem commented May 22, 2024

patkje75 commented May 22, 2024

patkje75 commented May 23, 2024

sebassem commented May 23, 2024 •

edited

Loading

sebassem commented May 23, 2024

MSBrett Jun 4, 2024

MSBrett Jun 4, 2024

MSBrett Jun 4, 2024

sebassem Jun 6, 2024 •

edited

Loading

MSBrett Jun 14, 2024

patkje75 Jun 14, 2024

MSBrett Jul 17, 2024

Added Private Endpoint deployment option #609

Are you sure you want to change the base?

Added Private Endpoint deployment option #609

Conversation

patkje75 commented Feb 19, 2024 • edited by flanakin Loading

🛠️ Description

🔬 How did you test this change?

🙋‍♀️ Do any of the following that apply?

📑 Did you update docs/changelog.md?

📖 Did you update documentation?

patkje75 commented Feb 19, 2024

flanakin left a comment

Choose a reason for hiding this comment

sebassem left a comment • edited Loading

Choose a reason for hiding this comment

bradlovas commented Apr 24, 2024

patkje75 commented Apr 25, 2024

patkje75 commented May 8, 2024

sebassem commented May 15, 2024

patkje75 commented May 22, 2024

sebassem commented May 22, 2024

patkje75 commented May 22, 2024

patkje75 commented May 23, 2024

sebassem commented May 23, 2024 • edited Loading

sebassem commented May 23, 2024

MSBrett Jun 4, 2024

Choose a reason for hiding this comment

MSBrett Jun 4, 2024

Choose a reason for hiding this comment

MSBrett Jun 4, 2024

Choose a reason for hiding this comment

sebassem Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

MSBrett Jun 14, 2024

Choose a reason for hiding this comment

patkje75 Jun 14, 2024

Choose a reason for hiding this comment

MSBrett Jul 17, 2024

Choose a reason for hiding this comment

patkje75 commented Feb 19, 2024 •

edited by flanakin

Loading

📑 Did you update `docs/changelog.md`?

sebassem left a comment •

edited

Loading

sebassem commented May 23, 2024 •

edited

Loading

sebassem Jun 6, 2024 •

edited

Loading