Skip to content

[IDP-42] DDS: Forcepoint Security Service Edge: Crawler Integration v1.0.0 #19360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open

[IDP-42] DDS: Forcepoint Security Service Edge: Crawler Integration v1.0.0 #19360

wants to merge 21 commits into from

Conversation

manan-crest
Copy link
Contributor

What does this PR do?

This is a initial release PR of Forcepoint Security Service Edge integration including all the required assets.

Motivation

  • Crawler code for this integration has been committed in its respective repository.
  • Pipeline and Facet group created for this integration are available in our sandbox and would be shared separately with the required teams.
  • Samples for the pipeline review would also be shared separately with the required teams.
  • OOTB detection rules JSON would be shared separately with the required teams as a part of separate repository.
  • Since during the standard attribute remapping we are not preserving the source attributes as per suggested best practices, it would result in filters using these standard attributes populating the values of other integrations as well as per current Datadog behaviour.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@manan-crest manan-crest force-pushed the forcepoint-security-service-edge-assets-v1.0.0 branch from 3fd524f to bff4198 Compare January 13, 2025 11:28
@manan-crest manan-crest marked this pull request as ready for review February 14, 2025 13:30
@manan-crest manan-crest requested review from a team as code owners February 14, 2025 13:30
@manan-crest manan-crest requested a review from a team February 14, 2025 13:30
@manan-crest manan-crest requested review from a team as code owners February 14, 2025 13:30
jnhunsberger
jnhunsberger previously requested changes Mar 7, 2025
View reviewed changes
Copy link

@jnhunsberger jnhunsberger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make requested copy change in the tile overview.

forcepoint_security_service_edge/README.md Outdated

This integration ingests the following logs:

- **Cloud Logs (CloudSummary, CloudAudit)**: Logs related to scanning results of each file.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloud Logs are "logs related to scanning results of each file"? That doesn't make sense. What files? Please update the text here to better reflect what is collected with these logs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have updated description to Logs related to the current status of files in cloud applications and scan results for each file in the account.

@manan-crest manan-crest requested a review from a team as a code owner March 10, 2025 12:25
@temporal-github-worker-1 temporal-github-worker-1 bot dismissed estherk15’s stale review March 10, 2025 12:25

Review from estherk15 is dismissed.
Related teams and files:

  • documentation
    • forcepoint_security_service_edge/README.md
nhinsch
nhinsch reviewed Mar 19, 2025
View reviewed changes
forcepoint_security_service_edge/assets/logs/forcepoint-security-service-edge.yaml Outdated
supportRules: ""
matchRules: syslog_header_rule <%{integer:syslog.priority}>%{integer}
%{notSpace} %{hostname:syslog.hostname} %{notSpace} %{notSpace}
%{notSpace:syslog.msgid}%{data}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a space before the final %{data}? While the parser probably works without it (since a space technically falls into data) I believe this would be more efficient if we added a space.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for suggestion! Based on observations from live logs, we expect the last field to be syslog.msgid. To ensure the parser remains stable even if there is any additional value after syslog.msgid, we have used %{data}.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK -- but I believe even in the situation you are describing, there will be a space after syslog.msgid. Therefore I suggest adding a space between %{notSpace:syslog.msgid} and %{data}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’ve decided to remove %{data} altogether and have made the necessary changes.

nhinsch
nhinsch reviewed Mar 19, 2025
View reviewed changes
forcepoint_security_service_edge/assets/logs/forcepoint-security-service-edge.yaml
enabled: true
filter:
query: source:forcepoint-security-service-edge
processors:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like these pipelines don't have status remappers defined. To confirm: are all of these logs to be considered INFO logs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not have any specific field for the status of an event. hence, we haven't created any status remapper.

@manan-crest manan-crest requested a review from nhinsch March 21, 2025 13:44
@nhinsch nhinsch added the assets/deploy-logs-staging ONLY USED BY Logs Backend - Validates that a PR is OK to go to staging label Apr 7, 2025
@torosmassa torosmassa dismissed jnhunsberger’s stale review April 21, 2025 13:25

Crest has addressed the feedback. Dismissing re-review/approval, so we can move to testing.

@Wyrine
Copy link
Contributor

Wyrine commented Apr 22, 2025

Please fix the conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nhinsch nhinsch nhinsch approved these changes

@Wyrine Wyrine Wyrine approved these changes

@estherk15 estherk15 estherk15 approved these changes

@jnhunsberger jnhunsberger Awaiting requested review from jnhunsberger

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
assets/deploy-logs-staging ONLY USED BY Logs Backend - Validates that a PR is OK to go to staging changelog/no-changelog docs/approved documentation ecosystems/approved integration/forcepoint_security_service_edge product/review-requested team/agent-integrations team/documentation team/logs-backend team/Logs-Core team/logs-integrations-reviewers team/saas-integrations team/siem-logs-reviewers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@manan-crest @ankitarajput-crest @shubhamvekariya-crest @Wyrine @nhinsch @jnhunsberger @estherk15