FRW-11191 Add robots.txt to block specific paths for all user agents

[vitaliiivanovspryker]

Overview

• Ticket: https://spryker.atlassian.net/browse/FRW-11191

Optional

• Core PR: URL_HERE

Change log

Added robots.txt to disallow sensitive and irrelevant paths

{Relevant changes for project level go here.}

CI Notice

Tests do not run automatically on every PR.
To trigger the required test suites, please add the appropriate label(s) to your PR.
For the full list of labels and their associated tests, see our Confluence page: CI Test Labels & Triggers

[snyk-spryker]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.