|
| 1 | +--- |
| 2 | +title: "Datavis in Babashka: analysing our calendar feed" |
| 3 | +author: |
| 4 | + - name: Daniel Slutsky |
| 5 | + url: https://scicloj.github.io/contributors/daslu/ |
| 6 | + image: https://avatars.githubusercontent.com/u/5673102?v=4 |
| 7 | + links: |
| 8 | + - {icon: github, href: 'https://github.com/daslu'} |
| 9 | +type: post |
| 10 | +date: '2026-05-02' |
| 11 | +category: data_analysis |
| 12 | +tags: [babashka, babqua, data, clojure-events] |
| 13 | +image: clojure-events-feed.png |
| 14 | +draft: true |
| 15 | +filters: |
| 16 | + - bb |
| 17 | +--- |
| 18 | + |
| 19 | +[Babashka](https://babashka.org) recently joined the set of languages supported on Clojure Civitas, thanks to [Babqua](https://github.com/scicloj/babqua) — a [Quarto](https://quarto.org) extension that evaluates `{.clojure .bb}` code blocks during render. The [introductory post](hello.html) covers the setup. Here we want to push a little further and see how far Babashka goes beyond its familiar home in scripting and shell automation — in particular, whether it works comfortably as a tool for interactive data analysis. |
| 20 | + |
| 21 | +This post is a small experiment in that direction. It is a Babashka-flavored adaptation of [a Noj tutorial](https://github.com/scicloj/noj-v2-getting-started) that walks through the same dataset using [tablecloth](https://scicloj.github.io/tablecloth/) and [tableplot](https://scicloj.github.io/tableplot/). Babashka does not ship those libraries, so we lean on what is already at hand: Clojure's core seq operations (`map`, `filter`, `group-by`, `sort-by`) over sequences of maps, plus a few of [Babashka's bundled facilities](https://book.babashka.org/#libraries) — `slurp` for fetching over HTTP, `clojure.string` for tokenizing, `java.time` for parsing timestamps — and [Plotly.js](https://plotly.com/javascript/) for the charts. The shape of the analysis is the same; only the toolbox is lighter. |
| 22 | + |
| 23 | +The early signs are promising. Writing data-analysis posts in Babashka with `quarto preview` watching the file feels close to a live notebook: each save re-renders within a second or two — fresh `bb` process, fresh data, refreshed plots — and that tight loop is what makes the workflow look genuinely viable for data analysis. Babashka's millisecond startup is doing most of the work. |
| 24 | + |
| 25 | +## The dataset |
| 26 | + |
| 27 | +The [Clojure events calendar feed](https://www.clojurians-zulip.org/feeds/events.ics) is an [ICS](https://en.wikipedia.org/wiki/ICalendar) file that lists meetups, conferences, study groups, and CFP deadlines from organizers across the Clojure community. It is maintained by [Gert Goet](https://github.com/eval), who started it in 2020 and has been steadily curating it since. Gert wrote a short [retrospective on Clojureverse](https://clojureverse.org/t/the-clojure-events-calendar-feed-turns-2/) when the feed turned two — that post is a good starting point if you want to understand what the feed is and how it grew. |
| 28 | + |
| 29 | +The feed is one of the few places where the rhythm of Clojure community activity is visible in one structured form. Reading it tells you about how user groups have been meeting, when conferences cluster in the calendar year, and how new initiatives appear and pick up. None of that information is hard to gather by hand, but having it as a single ICS file makes it tractable. |
| 30 | + |
| 31 | +A sincere thank-you to Gert for keeping the feed alive, and to the organizers whose events flow into it. |
| 32 | + |
| 33 | +## Setup |
| 34 | + |
| 35 | +Only the `java.time` classes need an explicit `import`. Babashka pre-aliases `clojure.string` as `str` in user code, so the familiar `str/split`, `str/replace`, and friends are already on hand without a `require`. |
| 36 | + |
| 37 | +```{.clojure .bb} |
| 38 | +(import '[java.time LocalDateTime] |
| 39 | + '[java.time.format DateTimeFormatter]) |
| 40 | +``` |
| 41 | + |
| 42 | +## Fetching the feed |
| 43 | + |
| 44 | +Babashka's `slurp` accepts URLs directly, so a single call retrieves the live feed. Each render of this post fetches it again; if you want to keep state warm across renders, see the [persistent-session option](https://scicloj.github.io/babqua/workflow.html) in the Babqua docs. |
| 45 | + |
| 46 | +```{.clojure .bb} |
| 47 | +(def feed-string |
| 48 | + (slurp "https://www.clojurians-zulip.org/feeds/events.ics")) |
| 49 | +
|
| 50 | +(count feed-string) |
| 51 | +``` |
| 52 | + |
| 53 | +A peek at the raw ICS — the first slice is enough to see the structure. The `^:kind/...` metadata used in the blocks below follows the [Kindly](https://scicloj.github.io/kindly-noted/kindly) convention for annotating values with how they should be rendered; Babqua's [kindly kinds page](https://scicloj.github.io/babqua/kindly-kinds.html) lists which kinds are currently supported. |
| 54 | + |
| 55 | +```{.clojure .bb} |
| 56 | +^:kind/hiccup |
| 57 | +[:pre {:style "max-height: 320px; overflow-y: auto; font-size: 12px;"} |
| 58 | + (subs feed-string 0 (min 1500 (count feed-string)))] |
| 59 | +``` |
| 60 | + |
| 61 | +Each event is wrapped in `BEGIN:VEVENT` / `END:VEVENT` and contains lines like `SUMMARY:`, `DTSTART:`, `URL:`. We only need those three fields. |
| 62 | + |
| 63 | +## Parsing |
| 64 | + |
| 65 | +We split the file on the boundary between events, then for each event we keep the lines that start with one of our three keys. The result of parsing one event is a small map; the result of parsing the file is a sequence of such maps. |
| 66 | + |
| 67 | +```{.clojure .bb} |
| 68 | +(defn parse-event |
| 69 | + "Return {:summary ... :dtstart ... :url ...} for a single VEVENT chunk." |
| 70 | + [event-text] |
| 71 | + (->> event-text |
| 72 | + str/split-lines |
| 73 | + (keep (fn [line] |
| 74 | + (when-let [field (re-find #"URL:|SUMMARY:|DTSTART:" line)] |
| 75 | + [(-> field (str/replace ":" "") str/lower-case keyword) |
| 76 | + (str/replace line field "")]))) |
| 77 | + (into {}))) |
| 78 | +
|
| 79 | +(def raw-events |
| 80 | + (->> (str/split feed-string #"END:VEVENT\nBEGIN:VEVENT") |
| 81 | + (map parse-event) |
| 82 | + (filter :dtstart))) |
| 83 | +
|
| 84 | +(count raw-events) |
| 85 | +``` |
| 86 | + |
| 87 | +A look at one parsed event: |
| 88 | + |
| 89 | +```{.clojure .bb} |
| 90 | +(first raw-events) |
| 91 | +``` |
| 92 | + |
| 93 | +## Enriching |
| 94 | + |
| 95 | +The `:dtstart` field is a string like `20240315T120000Z`. We parse it into a `LocalDateTime`, derive the year (handy for filtering), and keep an ISO-formatted string for charting (Plotly auto-detects date axes from ISO timestamps). |
| 96 | + |
| 97 | +```{.clojure .bb} |
| 98 | +(def ics-formatter |
| 99 | + (DateTimeFormatter/ofPattern "yyyyMMdd'T'HHmmss'Z'")) |
| 100 | +
|
| 101 | +(defn enrich [event] |
| 102 | + (let [datetime (LocalDateTime/parse (:dtstart event) ics-formatter)] |
| 103 | + (assoc event |
| 104 | + :datetime datetime |
| 105 | + :iso (str datetime) |
| 106 | + :year (.getYear datetime)))) |
| 107 | +
|
| 108 | +(def events |
| 109 | + (->> raw-events |
| 110 | + (map enrich) |
| 111 | + (sort-by :datetime))) |
| 112 | +
|
| 113 | +(count events) |
| 114 | +``` |
| 115 | + |
| 116 | +A small table of the first few rows, just to confirm the shape: |
| 117 | + |
| 118 | +```{.clojure .bb} |
| 119 | +^:kind/table |
| 120 | +{:column-names [:iso :summary :url] |
| 121 | + :row-vectors (->> events |
| 122 | + (take 5) |
| 123 | + (mapv (juxt :iso :summary :url)))} |
| 124 | +``` |
| 125 | + |
| 126 | +## A first plot |
| 127 | + |
| 128 | +Adding a running index gives us a y-coordinate. Each event is a point at `(date, total-events-so-far)`, so the line traces the feed's growth. |
| 129 | + |
| 130 | +```{.clojure .bb} |
| 131 | +(def with-cumulative-count |
| 132 | + (map-indexed (fn [index event] |
| 133 | + (assoc event :count (inc index))) |
| 134 | + events)) |
| 135 | +
|
| 136 | +^:kind/plotly |
| 137 | +{:data [{:type "scatter" |
| 138 | + :mode "lines+markers" |
| 139 | + :x (mapv :iso with-cumulative-count) |
| 140 | + :y (mapv :count with-cumulative-count) |
| 141 | + :text (mapv :summary with-cumulative-count) |
| 142 | + :hovertemplate "%{text}<br>%{x|%Y-%m-%d}<extra></extra>" |
| 143 | + :line {:width 1 :color "#5e81ac"} |
| 144 | + :marker {:size 5 :color "#5e81ac"}}] |
| 145 | + :layout {:xaxis {:title "date"} |
| 146 | + :yaxis {:title "events so far"} |
| 147 | + :width 740 |
| 148 | + :height 340 |
| 149 | + :margin {:t 20 :r 20}}} |
| 150 | +``` |
| 151 | + |
| 152 | +The plot shows a noticeable gap around 2022 — a stretch where the feed has very few entries. The reasons are historical and beyond the scope of this analysis; for the questions we want to ask here (which groups have been active, and how often they have been meeting), the cleaner thing to do is to restrict attention to 2023 and later, where the feed is densely populated. |
| 153 | + |
| 154 | +## Recent years |
| 155 | + |
| 156 | +```{.clojure .bb} |
| 157 | +(def recent-events |
| 158 | + (filter #(>= (:year %) 2023) with-cumulative-count)) |
| 159 | +
|
| 160 | +(count recent-events) |
| 161 | +``` |
| 162 | + |
| 163 | +The cumulative count is preserved, so the y-axis still measures global "events so far" — we are simply zooming in on the right-hand portion of the timeline. |
| 164 | + |
| 165 | +```{.clojure .bb} |
| 166 | +^:kind/plotly |
| 167 | +{:data [{:type "scatter" |
| 168 | + :mode "lines+markers" |
| 169 | + :x (mapv :iso recent-events) |
| 170 | + :y (mapv :count recent-events) |
| 171 | + :text (mapv :summary recent-events) |
| 172 | + :hovertemplate "%{text}<br>%{x|%Y-%m-%d}<extra></extra>" |
| 173 | + :line {:width 1 :color "#5e81ac"} |
| 174 | + :marker {:size 5 :color "#5e81ac"}}] |
| 175 | + :layout {:xaxis {:title "date"} |
| 176 | + :yaxis {:title "events so far"} |
| 177 | + :width 740 |
| 178 | + :height 340 |
| 179 | + :margin {:t 20 :r 20}}} |
| 180 | +``` |
| 181 | + |
| 182 | +## Recognizing groups |
| 183 | + |
| 184 | +Many event URLs contain a slug that identifies the organizing group — `london-clojurians`, `los-angeles-clojure`, `data-recur`, and so on. A small alternation regex covers the groups whose names appear frequently in the feed; everything else falls into an `"other"` bucket. |
| 185 | + |
| 186 | +```{.clojure .bb} |
| 187 | +(def groups-pattern |
| 188 | + #"london-clojurians|los-angeles-clojure|visual-tools|data-recur|real-world-data|scicloj-llm|scicloj-ai|macroexpand|clojure-dsp") |
| 189 | +
|
| 190 | +(defn assign-group [event] |
| 191 | + (assoc event :group |
| 192 | + (or (some->> (:url event) str/lower-case (re-find groups-pattern)) |
| 193 | + "other"))) |
| 194 | +
|
| 195 | +(def grouped-events |
| 196 | + (->> recent-events |
| 197 | + (filter :url) |
| 198 | + (map assign-group))) |
| 199 | +
|
| 200 | +(->> grouped-events |
| 201 | + (map :group) |
| 202 | + frequencies |
| 203 | + (sort-by key)) |
| 204 | +``` |
| 205 | + |
| 206 | +## Coloring by group |
| 207 | + |
| 208 | +Re-using the global `:count` and adding `:group` as a color channel: each group's events scatter along the same growth curve, and the colors give a sense of how the community's activity has been spread over time. |
| 209 | + |
| 210 | +```{.clojure .bb} |
| 211 | +^:kind/plotly |
| 212 | +{:data (->> (group-by :group grouped-events) |
| 213 | + (sort-by key) |
| 214 | + (mapv (fn [[group events-in-group]] |
| 215 | + {:type "scatter" |
| 216 | + :mode "markers" |
| 217 | + :name group |
| 218 | + :x (mapv :iso events-in-group) |
| 219 | + :y (mapv :count events-in-group) |
| 220 | + :text (mapv :summary events-in-group) |
| 221 | + :hovertemplate "%{text}<br>%{x|%Y-%m-%d}<extra>%{fullData.name}</extra>" |
| 222 | + :marker {:size 7}}))) |
| 223 | + :layout {:xaxis {:title "date"} |
| 224 | + :yaxis {:title "events so far"} |
| 225 | + :width 780 |
| 226 | + :height 400 |
| 227 | + :legend {:title {:text "group"}} |
| 228 | + :margin {:t 20 :r 20}}} |
| 229 | +``` |
| 230 | + |
| 231 | +## A per-group time series |
| 232 | + |
| 233 | +The previous plot uses a single global counter, so each group's points lie on the overall curve. To follow how each group has been evolving on its own — when it picked up, when it went quiet, when something new started — we want a per-group running count: how many events that group has held up to each date. In tablecloth this is `group-by` + `add-column` + `ungroup`; in plain Clojure it is `group-by` + `mapcat` over the partitions. |
| 234 | + |
| 235 | +```{.clojure .bb} |
| 236 | +(def per-group-series |
| 237 | + (->> grouped-events |
| 238 | + (group-by :group) |
| 239 | + (mapcat (fn [[_ events-in-group]] |
| 240 | + (->> events-in-group |
| 241 | + (sort-by :datetime) |
| 242 | + (map-indexed (fn [index event] |
| 243 | + (assoc event :group-count (inc index))))))))) |
| 244 | +
|
| 245 | +(->> per-group-series |
| 246 | + (filter #(= "london-clojurians" (:group %))) |
| 247 | + (take 3) |
| 248 | + (map #(select-keys % [:iso :group :group-count :summary]))) |
| 249 | +``` |
| 250 | + |
| 251 | +Layering points on top of lines gives a per-group trajectory. Each line tells its own story — periods of momentum, pauses, fresh starts — and together they sketch where the community has been heading. |
| 252 | + |
| 253 | +```{.clojure .bb} |
| 254 | +^:kind/plotly |
| 255 | +{:data (->> (group-by :group per-group-series) |
| 256 | + (sort-by key) |
| 257 | + (mapv (fn [[group events-in-group]] |
| 258 | + {:type "scatter" |
| 259 | + :mode "lines+markers" |
| 260 | + :name group |
| 261 | + :x (mapv :iso events-in-group) |
| 262 | + :y (mapv :group-count events-in-group) |
| 263 | + :text (mapv :summary events-in-group) |
| 264 | + :hovertemplate "%{text}<br>%{x|%Y-%m-%d}<extra>%{fullData.name}</extra>" |
| 265 | + :line {:width 1.5} |
| 266 | + :marker {:size 6}}))) |
| 267 | + :layout {:xaxis {:title "date"} |
| 268 | + :yaxis {:title "events per group"} |
| 269 | + :width 780 |
| 270 | + :height 440 |
| 271 | + :legend {:title {:text "group"}} |
| 272 | + :margin {:t 20 :r 20}}} |
| 273 | +``` |
| 274 | + |
| 275 | +The `real-world-data` line in particular is worth pointing at. The [Real-World Data dev group](https://scicloj.github.io/docs/community/groups/real-world-data/), recently reinitiated by [Timothy Pratley](https://github.com/timothypratley), has been meeting on a steady weekly rhythm — each point on that line is a session that took preparation, hosting, and follow-through. The slope is a record of persistent work, and a thank-you is due to Tim and the participants who have kept it going. |
| 276 | + |
| 277 | +## Closing notes |
| 278 | + |
| 279 | +The substance of this analysis lives in the feed itself, not in the code that reads it. The feed exists because Gert decided to maintain it and because organizers around the Clojure world keep listing their events. Counting and plotting are easy; the curation that produces the underlying data is the work that makes the rest possible. |
| 280 | + |
| 281 | +If you organize a Clojure event and it is not yet in the feed, the [Clojureverse thread](https://clojureverse.org/t/the-clojure-events-calendar-feed-turns-2/) is a good place to start; submitting an event is a single CLI call. |
| 282 | + |
| 283 | +If you are curious about Babashka for data analysis — or interested in how far this lighter toolbox can be pushed — we would be glad to hear from you. The [Real-World Data dev group](https://scicloj.github.io/docs/community/groups/real-world-data/) is one of the spaces where this kind of exploration is happening, and visitors and collaborators are welcome. |
| 284 | + |
| 285 | +For more on writing Babashka posts on Civitas, see the [introductory post](hello.html) and the [Babqua docs](https://scicloj.github.io/babqua/). |
0 commit comments