Skip to content

Generate Hadoop DSL visualizations for workflow graphs #113

@convexquad

Description

@convexquad

Internally at LinkedIn Luke Duncan created the JIRA LIHADOOP-20359 with the following message:

Hey Hadoop Team,

I find myself often taking screenshots of Azkaban DAGs to include them in
RBs. Reading Hadoop DSL refactorings in RB can sometimes be difficult and
the visual help. I'm pretty sure the open source version of Azkaban
includes the tool for visualizing Azkaban DAGs. I wonder if we could use it
in git-review, and automatically create a new image and include it with
every RB that affects a hadoop multi-product?

    Luke

My response was:

Luke, a really cool way to do this would be to add an implementation of the Hadoop DSL Compiler class (https://github.com/linkedin/linkedin-gradle-plugin-for-apache-hadoop/blob/master/hadoop-plugin/src/main/groovy/com/linkedin/gradle/hadoopdsl/HadoopDslCompiler.groovy) that generates a GraphViz DOT file, and then add a Gradle task to the Hadoop Plugin that runs the compiler and directly generates a PNG based on your DOT file.

I still think this would be an interesting and useful Hadoop DSL enhancement.

-Alex

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions