[Web] Setup header for web scrape task

[ShihChun-H]

Issue Description

Current State

• Web crawler cannot set header auth

Why We Want to Change?

• With header auth, the users can crawl the web requires auth.
• It gives VDP more use cases.

Proposed Change

• Setup input for all tasks in Web operator
  "headers": {}
• Users can set auth and content type

---

Rules for the Component Hackathon

• Each issue will only be assigned to one person/team at a time.
• You can only work on one issue at a time.
• To express interest in an issue, please comment on it and tag @kuroxx, allowing the Instill AI team to assign it to you.
• Ensure you address all feedback and suggestions provided by the Instill AI team.
• If no commits are made within five days, the issue may be reassigned to another contributor.
• Join our Discord to engage in discussions and seek assistance in #hackathon channel. For technical queries, you can tag @chuang8511.

---

Component Contribution Guideline | Documentation | Official Go Tutorial

[linear]

INS-6353 [Web] Setup header for web scrape task

[someshfengde]

Hi @ShihChun-H can you please assign this issue to me?

[ShihChun-H]

Hi @someshfengde , sure. The issue has been assigned to you.