How can I integrate Superset charts with PlayWright to ensure that all charts and data on the page are fully rendered and there are no unexpected errors before taking a screenshot?

[1sanqian]

How can I integrate Superset charts with PlayWright to ensure that all charts and data on the page are fully rendered and there are no unexpected errors before taking a screenshot?I'm using C#, and I'm using Supersest version 4.0.1.

[dosubot[bot]]

To ensure all Superset charts and data are fully rendered and error-free before taking a screenshot with Playwright in C#, follow the same logic used by Superset's built-in Playwright automation:

1. Navigate to your Superset dashboard or chart URL.
2. Wait for the main container (e.g., .dashboard or the relevant class) to appear.
3. Wait for all .chart-container elements to be visible.
4. Wait for all .loading elements to disappear (state "detached"), which means charts have finished rendering.
5. (Optional) Wait a few seconds for chart animations to complete.
6. Check for error elements with role="alert" and extract error messages from .antd5-modal-content .antd5-modal-body if present.
7. Take the screenshot.

Here’s a C# Playwright code snippet that implements this logic:

using Microsoft.Playwright;

using var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions { Headless = true });
var page = await browser.NewPageAsync();

await page.GotoAsync("https://your-superset-url/superset/dashboard/123/", new PageGotoOptions { WaitUntil = WaitUntilState.Load });

// Wait for main dashboard container
await page.WaitForSelectorAsync(".dashboard");

// Wait for all chart containers to be visible
var chartContainers = await page.QuerySelectorAllAsync(".chart-container");
foreach (var chart in chartContainers)
{
    await chart.WaitForElementStateAsync(ElementState.Visible);
}

// Wait for all loading spinners to disappear
var loadingElements = await page.QuerySelectorAllAsync(".loading");
foreach (var loading in loadingElements)
{
    await loading.WaitForElementStateAsync(ElementState.Detached);
}

// Optional: wait for chart animations to finish
await page.WaitForTimeoutAsync(5000);

// Check for unexpected errors
var alerts = await page.QuerySelectorAllAsync("[role='alert']");
foreach (var alert in alerts)
{
    var seeMoreBtn = await alert.QuerySelectorAsync("[role='button']");
    if (seeMoreBtn != null)
    {
        await seeMoreBtn.ClickAsync();
        var modalBody = await page.QuerySelectorAsync(".antd5-modal-content .antd5-modal-body");
        if (modalBody != null)
        {
            var errorMsg = await modalBody.InnerTextAsync();
            // Handle or log errorMsg as needed
        }
        var closeBtn = await page.QuerySelectorAsync(".antd5-modal-content .antd5-modal-close");
        if (closeBtn != null)
            await closeBtn.ClickAsync();
        await page.WaitForSelectorAsync(".antd5-modal-content", new PageWaitForSelectorOptions { State = WaitForSelectorState.Detached });
    }
}

// Take the screenshot
await page.ScreenshotAsync(new PageScreenshotOptions { Path = "dashboard.png" });

For large dashboards, consider a tiled screenshot approach: scroll the dashboard in view-sized increments, wait for content to settle after each scroll, and stitch the images together. This is how Superset handles dashboards that extend beyond the viewport [source].

You can tune timeouts and delays using Superset config options like SCREENSHOT_PLAYWRIGHT_DEFAULT_TIMEOUT, SCREENSHOT_SELENIUM_ANIMATION_WAIT, and SCREENSHOT_SELENIUM_HEADSTART [source].

There are no C#-specific examples in the Superset codebase, but the DOM structure and selectors are the same as in Python, so this approach is portable [source].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dosubot[bot]]

To address 502 Bad Gateway errors in Superset caused by high server load, network issues, or timeouts, focus on these areas:

1. Increase timeouts and enable retries:

• In superset_config.py, raise SUPERSET_WEBSERVER_TIMEOUT (e.g., to 300 seconds), SQLLAB_TIMEOUT, and SQLLAB_ASYNC_TIME_LIMIT_SEC for long queries.
• For screenshot automation, set SCREENSHOT_LOCATE_WAIT and SCREENSHOT_LOAD_WAIT to higher values (e.g., 120–600 seconds) to give charts more time to render before timing out [example config].
• Enable and tune Superset's HTTP retry logic for transient 502/503/504 errors:

  SUPERSET_CLIENT_RETRY_ATTEMPTS = 3
  SUPERSET_CLIENT_RETRY_DELAY = 1000  # ms
  SUPERSET_CLIENT_RETRY_BACKOFF_MULTIPLIER = 2
  SUPERSET_CLIENT_RETRY_MAX_DELAY = 10000  # ms
  SUPERSET_CLIENT_RETRY_JITTER_MAX = 1000  # ms
  SUPERSET_CLIENT_RETRY_STATUS_CODES = [502, 503, 504]

  This helps Superset automatically retry failed chart data requests with exponential backoff and jitter [details].

2. Tune your proxy and web server:

• In Nginx, increase proxy_read_timeout, proxy_connect_timeout, and proxy_send_timeout to be higher than your Superset/Gunicorn timeouts to avoid premature 502/504 errors [discussion].
• For Gunicorn, increase the keepalive setting to handle more concurrent users and avoid ECONNRESET errors that can manifest as 502s [discussion].

3. Scale and monitor resources:

• Make sure your Superset server and database have enough CPU and RAM. 8GB RAM and 2vCPUs is a baseline for moderate use, but usage spikes (e.g., many users loading dashboards at once) may require more [faq].
• Monitor server, database, and network health. Look for slow queries, high CPU/memory usage, or network bottlenecks.

4. Optimize queries and database performance:

• Slow or complex queries are a common cause of timeouts and 502 errors. Optimize your dashboards and underlying SQL, and consider using a more performant database engine if needed [faq].

5. Use asynchronous query execution:

• Enable async queries in Superset (via Celery) to allow long-running queries to complete without blocking the web server, reducing the risk of timeouts and 502 errors [async config].

6. Troubleshoot persistent errors:

• Check Superset, database, and proxy logs for errors or timeouts.
• Verify network connectivity between Superset, the database, and any proxies.
• If using Docker/Kubernetes, ensure the worker containers have access to the Superset instance and required headless browsers for screenshots.

These steps should help reduce the frequency of 502 errors and improve the reliability of chart rendering and screenshot automation in Superset.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[1sanqian]

I've already determined the page has rendered based on the message "Wait for all .loading elements to disappear (status is 'detached'), which means the chart has finished rendering." However, there's a brief period where the .loading elements disappear but the chart hasn't appeared yet. How can I fix this?@dosu

[dosubot[bot]]

This happens because Superset removes the .loading spinner as soon as the data fetch finishes, but the chart's actual rendering (drawing SVG, canvas, or DOM updates) can take a bit longer due to asynchronous UI updates and animations. So, waiting only for .loading to disappear isn't always enough.

To make your Playwright automation more robust, you should:

• Wait for all .chart-container elements to be visible.
• Wait for all .loading elements to be detached.
• Then, for each chart, wait for a specific chart content element (like #chart-id-{chartId}) to be visible—this element only appears when the chart is fully rendered.
• Optionally, add a short extra wait (e.g., 1–2 seconds) for chart animations to finish. Superset's own screenshot logic uses a configurable animation wait for this reason.

Example C# Playwright logic:

// Wait for all chart containers to be visible
var chartContainers = await page.QuerySelectorAllAsync(".chart-container");
foreach (var chart in chartContainers)
{
    await chart.WaitForElementStateAsync(ElementState.Visible);
}

// Wait for all loading spinners to disappear
var loadingElements = await page.QuerySelectorAllAsync(".loading");
foreach (var loading in loadingElements)
{
    await loading.WaitForElementStateAsync(ElementState.Detached);
}

// Wait for each chart's content to be visible (replace with actual chart IDs)
foreach (var chart in chartContainers)
{
    var chartId = await chart.GetAttributeAsync("data-test-chart-id");
    if (!string.IsNullOrEmpty(chartId))
    {
        await page.WaitForSelectorAsync($"#chart-id-{chartId}", new PageWaitForSelectorOptions { State = WaitForSelectorState.Visible });
    }
}

// Optional: wait for chart animations to finish
await page.WaitForTimeoutAsync(1000); // 1 second

This approach matches how Superset's own automation and Cypress tests ensure charts are fully rendered before screenshots or assertions. You can tune the extra wait with the SCREENSHOT_SELENIUM_ANIMATION_WAIT config if needed. More details:
Superset Playwright/Selenium screenshot logic
Why waiting for .loading isn't enough
ChartRenderer rendering lifecycle

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[1sanqian]

Could you please take a look at this code and see if it can effectively solve the problems of server load and complete screenshots/charts?@dosu.

private async Task<(bool isValid, string reason)> PerformFinalChartValidationAsync(IPage page, string originalTemplateUrl)
{
var validationStartTime = DateTime.UtcNow;
var maxValidationTime = TimeSpan.FromMinutes(5);
var validationAttempt = 0;

    Log.Information("Starting 5-minute final chart validation cycle...");

    while (DateTime.UtcNow - validationStartTime < maxValidationTime)
    {
        validationAttempt++;
        var elapsed = DateTime.UtcNow - validationStartTime;

        Log.Information("Final validation attempt {Attempt} at {Elapsed:F1} minutes", validationAttempt, elapsed.TotalMinutes);

        try
        {
            Log.Information("Step 1: Waiting for main dashboard container...");
            await page.WaitForSelectorAsync(".dashboard, [class*='dashboard'], .main-content, [class*='main']",
                new PageWaitForSelectorOptions { Timeout = 10000 }).ConfigureAwait(false);

            Log.Information("Dashboard container found");

            Log.Information("Step 2: Waiting for all chart containers...");
            var chartContainers = await page.QuerySelectorAllAsync(".chart-container, [class*='chart-container'], [data-test*='chart'], .viz-chart");
            Log.Information("Found {Count} chart containers", chartContainers.Count);

        var visibleContainers = 0;
        var failedContainers = new List<string>();

        foreach (var container in chartContainers)
        {
            try
            {
                await container.WaitForElementStateAsync(ElementState.Visible, new ElementHandleWaitForElementStateOptions { Timeout = 30000 });
                visibleContainers++;
            }
            catch (Exception ex)
            {
                var containerInfo = await container.EvaluateAsync<string>("element => element.outerHTML.substring(0, 200)").ConfigureAwait(false);
                failedContainers.Add($"Container failed to become visible: {containerInfo}");
                Log.Information(ex, "Chart container failed to become visible within timeout");
            }
        }

        Log.Information("Chart containers visibility check: {Visible}/{Total} visible", visibleContainers, chartContainers.Count);

            if (failedContainers.Count > 0)
            {
                Log.Warning("Some chart containers failed to become visible: {Failed}/{Total}",
                    failedContainers.Count, chartContainers.Count);

                if (failedContainers.Count > chartContainers.Count * 0.5)
                {
                    var pageDiagnostics = await GatherPageDiagnosticsAsync(page);
                    throw new Exception($"过多图表容器不可见 ({failedContainers.Count}/{chartContainers.Count}): {string.Join("; ", failedContainers)}. 页面诊断: {pageDiagnostics}");
                }
            }
            Log.Information("All chart containers are visible");

        Log.Information("Step 3: Waiting for all loading elements to disappear (SCREENSHOT_LOAD_WAIT style)...");
        var loadingSelectors = new[]
        {
            ".loading",
            ".spinner",
            ".loader",
            "[class*='loading']",
            "[class*='is-loading']",
            "[class*='css-19o2u55']",
            "[class*='spinner']",
            "[class*='ant-spin']",
            ".anticon-loading"
        };

        var startTime = DateTime.UtcNow;
        var maxWaitTime = TimeSpan.FromSeconds(60);

        while (DateTime.UtcNow - startTime < maxWaitTime)
        {
            var hasLoadingElements = false;

            foreach (var selector in loadingSelectors)
            {
                try
                {
                    var loadingElements = await page.QuerySelectorAllAsync(selector);
                    foreach (var loadingElement in loadingElements)
                    {
                        var isAttached = await loadingElement.EvaluateAsync<bool>("element => document.contains(element)");
                        if (isAttached)
                        {
                            var isVisible = await loadingElement.IsVisibleAsync();
                            if (isVisible)
                            {
                                hasLoadingElements = true;
                                break;
                            }
                        }
                    }
                    if (hasLoadingElements) break;
                }
                catch
                {
                }
            }

            if (!hasLoadingElements)
            {
                Log.Information("All loading elements have disappeared (detached)");
                break;
            }

            await page.WaitForTimeoutAsync(500);
        }
        
        Log.Information("Step 4: Waiting for each chart's content to be visible...");
        
        var chartContentValidationErrors = new List<string>();
        foreach (var chart in chartContainers)
        {
            try
            {
                var chartId = await chart.GetAttributeAsync("data-test-chart-id") ??
                             await chart.GetAttributeAsync("data-chart-id") ??
                             await chart.GetAttributeAsync("id");

                if (!string.IsNullOrEmpty(chartId))
                {
                    await page.WaitForSelectorAsync($"#chart-id-{chartId}", new PageWaitForSelectorOptions
                    {
                        State = WaitForSelectorState.Visible,
                        Timeout = 10000
                    });
                    Log.Information("Chart content visible for chart ID: {ChartId}", chartId);
                }
                else
                {
                    await chart.WaitForSelectorAsync("svg, canvas, .chart-content, [class*='chart']", new ElementHandleWaitForSelectorOptions
                    {
                        State = WaitForSelectorState.Visible,
                        Timeout = 10000
                    });
                    Log.Information("Chart content visible for container (fallback method)");
                }
            }
            catch (Exception ex)
            {
                var containerInfo = await chart.EvaluateAsync<string>("element => element.outerHTML.substring(0, 200)").ConfigureAwait(false);
                chartContentValidationErrors.Add($"Chart content validation failed: {containerInfo}");
                Log.Warning(ex, "Failed to validate chart content visibility for container");
            }
        }

        if (chartContentValidationErrors.Count > 0)
        {
            Log.Warning("Some charts failed content validation: {Count}/{Total}",
                chartContentValidationErrors.Count, chartContainers.Count);

            if (chartContentValidationErrors.Count > chartContainers.Count * 0.3) // 30% threshold
            {
                throw new Exception($"过多图表内容未正确渲染 ({chartContentValidationErrors.Count}/{chartContainers.Count}): {string.Join("; ", chartContentValidationErrors)}");
            }
        }

        Log.Information("Step 5: Waiting for animation delay and headstart (Superset SCREENSHOT_SELENIUM_ANIMATION_WAIT)...");

        var animationWaitMs = 2000;

        var headstartMs =  500;

        var totalWaitMs = animationWaitMs + headstartMs;
        await page.WaitForTimeoutAsync(totalWaitMs);
        Log.Information("Animation and headstart delay completed ({AnimationMs}ms + {HeadstartMs}ms = {TotalMs}ms) - Superset timing parameters",
            animationWaitMs, headstartMs, totalWaitMs);
        
        var remainingLoadingElements = 0;
        foreach (var selector in loadingSelectors)
        {
            try
            {
                var count = await page.Locator(selector).CountAsync();
                remainingLoadingElements += count;
            }
            catch
            {
            }
        }

        if (remainingLoadingElements > 0)
        {
            Log.Information("Warning: {Count} loading elements still present after validation", remainingLoadingElements);
            if (remainingLoadingElements <= 3)
            {
                Log.Information("Accepting validation with {Count} remaining loading elements (likely minor UI elements)", remainingLoadingElements);
            }
            else
            {
                Log.Information("Accepting validation with {Count} remaining loading elements (likely minor UI elements)", remainingLoadingElements);
                return (false, $"TimeOut");
            }
        }

        Log.Information("Step 6: Checking for error alerts and attempting recovery...");
        var (errorMessages, recoveryAttempted) = await CheckAndHandleErrorAlertsWithRecoveryAsync(page, chartContainers);
        if (errorMessages.Count > 0)
        {
            Log.Warning("Found {Count} error alerts on page", errorMessages.Count);
            foreach (var errorMsg in errorMessages)
            {
                Log.Warning("Error alert: {Message}", errorMsg);
            }

            var criticalErrors = errorMessages.Where(msg =>
                msg.Contains("502") || msg.Contains("500") ||
                msg.Contains("Bad Gateway") ||
                msg.Contains("Connection failed") || msg.Contains("Network error")).ToList();

            if (criticalErrors.Count > 0)
            {
                var temporaryErrors = criticalErrors.Where(msg =>
                    msg.Contains("502") || msg.Contains("503") || msg.Contains("504") ||
                    msg.Contains("Bad Gateway") || msg.Contains("Service Unavailable") ||
                    msg.Contains("Gateway Timeout") ||
                    msg.Contains("Connection failed") || msg.Contains("Network error")).ToList();

                var permanentErrors = criticalErrors.Except(temporaryErrors).ToList();

                Log.Information("Detected temporary server errors, will refresh and retry...");
                throw new Exception($"临时服务器错误: {string.Join("; ", temporaryErrors)}");
            }
            else
            {
                if (recoveryAttempted)
                {
                    Log.Information("Recovery attempted for non-critical errors, waiting for charts to reload...");

                    await page.WaitForTimeoutAsync(3000);

                    var remainingErrors = await CheckForRemainingErrorsAsync(page);
                    if (remainingErrors.Count > 0)
                    {
                        Log.Information("Some errors persist after recovery attempt @{error}", string.Join("; ", remainingErrors));
                        return (false, $"TimeOut");
                    }
                    else
                    {
                        Log.Information("Chart recovery successful, proceeding with screenshot");
                    }
                }
                else
                {
                    Log.Information("Non-critical errors found but no recovery attempted, proceeding with screenshot");
                }
            }
        }

        var totalChartContainers = chartContainers.Count;
        Log.Information("Final validation: {TotalCharts} chart containers validated", totalChartContainers);

            if (totalChartContainers == 0)
            {
                throw new Exception("未找到任何图表容器");
            }
            
            Log.Information("Step 7: Final stability check...");
        var isStable = await page.EvaluateAsync<bool>(@"
            () => {
                // 检查页面是否还在滚动或有动画
                return !window.scrolling && document.readyState === 'complete';
            }
        ").ConfigureAwait(false);

        if (!isStable)
        {
            Log.Information("Page not fully stable, waiting additional time...");
            await page.WaitForTimeoutAsync(1000);
        }

        Log.Information("Superset-style validation completed successfully");
        
        Log.Information("Final validation passed, executing page preparation steps for screenshot...");

        await ScrollToBottomWithPlaywrightAsync(page).ConfigureAwait(false);

        await WaitForPageStableWithPlaywrightAsync(page, maxWaitSeconds: 3).ConfigureAwait(false);

        try
        {
            var collapseButton = page.Locator("button[data-test='filter-bar__collapse-button']");

            await collapseButton.WaitForAsync(new LocatorWaitForOptions
            {
                State = WaitForSelectorState.Visible,
                Timeout = 10000
            }).ConfigureAwait(false);

            await collapseButton.WaitForAsync(new LocatorWaitForOptions
            {
                State = WaitForSelectorState.Attached,
                Timeout = 10000
            }).ConfigureAwait(false);

            await collapseButton.ClickAsync(new LocatorClickOptions
            {
                Timeout = 10000
            }).ConfigureAwait(false);

            await page.WaitForTimeoutAsync(500);
            Log.Information("Successfully clicked collapse button after final validation");
        }
        catch (Exception ex)
        {
            Log.Information(ex, "Failed to click collapse button after final validation");
        }

        await ClearHoverStateWithPlaywrightAsync(page).ConfigureAwait(false);

            Log.Information("Page preparation completed after final validation");

            return (true, "Superset样式验证通过");
        }
        catch (Exception ex)
        {
            Log.Information("Validation attempt {Attempt} failed: {Error}", validationAttempt, ex.Message);

            if (DateTime.UtcNow - validationStartTime < maxValidationTime)
            {
                Log.Information("Refreshing page and retrying validation (attempt {Attempt})...", validationAttempt + 1);
                try
                {
                    await page.ReloadAsync(new PageReloadOptions { WaitUntil = WaitUntilState.DOMContentLoaded });
                    await page.WaitForTimeoutAsync(2000);
                    continue;
                }
                catch (Exception reloadEx)
                {
                    Log.Information(reloadEx, "Failed to reload page for retry");
                    continue;
                }
            }
            else
            {
                var totalElapsed = DateTime.UtcNow - validationStartTime;
                Log.Information("Final chart validation timeout after {Elapsed:F1} minutes and {Attempts} attempts",
                    totalElapsed.TotalMinutes, validationAttempt);
                return (false, "TimeOut");
            }
        }
    }

    var finalElapsed = DateTime.UtcNow - validationStartTime;
    Log.Error("Final chart validation timeout after {Elapsed:F1} minutes", finalElapsed.TotalMinutes);
    return (false, "TimeOut");
}

[dosubot[bot]]

Your validation method closely matches Superset’s own Playwright/Selenium screenshot automation and covers all the critical steps: waiting for the dashboard and chart containers, waiting for loading elements to disappear, checking for chart content, waiting for animation delays, detecting error alerts (including 502/503/504), and retrying on temporary failures. This is the recommended approach for ensuring charts are fully rendered before screenshots and for handling transient server load issues in Superset (reference).

However, no frontend automation can fully “solve” backend server load or persistent 502 errors—your code can only detect, retry, and fail gracefully if the backend cannot deliver data. For best results, make sure your Superset backend is tuned: increase timeouts like SCREENSHOT_LOCATE_WAIT and SCREENSHOT_LOAD_WAIT, enable HTTP retries (SUPERSET_CLIENT_RETRY_ATTEMPTS etc.), and optimize your database and proxy settings (reference). For very large dashboards, consider increasing your max wait times and resource limits.

If you want to go even further, you could check for the presence of login screens (to catch authentication/session issues), and monitor resource usage on your Superset server during heavy loads. But overall, your method is robust and aligns with Superset’s best practices for screenshot and chart completeness validation.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}