Skip to content

[Question] Are webworkers taken into account? #78

Open
@Jogai

Description

@Jogai

So I'm loading a certain page, but the actual data that I want to scrape is set by a script trough a webworker. It listens to a stream on a different endpoint to get the actual values.

Question1: Are webworkers supposed to run? (probably not).

Question2 if answer#1 is no: Is it possible to inject the script to the page to try and execute it.

Question3 if answer#2 is no: what would need to be done to make webworkers run? I might take a shot at it.

For the curious, my code so far:

            IConfiguration angleSharpConfig = Configuration.Default
                    .WithDefaultLoader(new LoaderOptions { IsResourceLoadingEnabled = true })
                    .WithDefaultCookies().WithJs();

            try
            {
                string cookieValue = String.Empty;
                IBrowsingContext angleSharpContext = BrowsingContext.New(angleSharpConfig);

                    string baseUrl = Regex.Match(sensor.Url, @"^https?:\/+([\d+|\.])+\/").Groups[0].ToString();
                    string loginUrl = $@"{baseUrl}cgi/login?username={usr}&password={pass}";

                    using (WebClient client = new WebClient())
                    {
                        string loginInfo = client.DownloadString(loginUrl);
                        cookieValue = loginInfo.Split(new[] { "\n", Environment.NewLine },
                                StringSplitOptions.RemoveEmptyEntries)
                            .FirstOrDefault(sx => sx.Contains("="));
                        angleSharpContext.SetCookie(new Url(baseUrl), cookieValue);
                    }

                IDocument document = await angleSharpContext.OpenAsync(sensor.Url).WaitUntilAvailable();

                IHtmlCollection<IElement> t = document.QuerySelectorAll("#m1_page1 [data-sid]");
                //At this point the elements have default values and not yet set by the script despite using WaitUntilAvailable
            }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions