You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _units/guide-collecting-posts-communalytic.md
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ authors:
14
14
- hannah-smyth
15
15
editors:
16
16
- sofia-papastamkou
17
-
date: 2025-10-06
17
+
publication_date: 2025-10-06
18
18
toc: off
19
19
mediatypes:
20
20
- web-social-media
@@ -54,13 +54,13 @@ The Bluesky API allows us to access one of the core primitives of the AT Protoco
54
54
55
55
Based on this API, the EDU version of Communalytic allows users to collect up to 30 000 Bluesky posts (including replies) in three datasets. Regardless of the version you use, Communalytic will take into account the Bluesky API [rate limits](https://docs.bsky.app/docs/advanced-guides/rate-limits), which are implemented to avoid service disruptions caused by intense data harvesting. This is all the more important with the rise of large language models and the urgent need for fresh data to train them, as more and more companies are now aggressively collecting content. This has led to a number of controversies, including some involving [Bluesky](https://www.404media.co/someone-made-a-dataset-of-one-million-bluesky-posts-for-machine-learning-research/). It is important to be aware of this issue, as it affects our privacy.
## Your first harvest of posts <aid="section3"></a>
60
60
61
61
To create a free EDU account in Communalytic, you need to sign in through Google. (Researchers working on larger projects can also create a PRO account with an email address, but for this tutorial we use the free version of the software.) Once you are connected, you will be directed to “My Datasets” (or click “My Datasets” in the main menu bar). You do not need to have a Bluesky account to harvest posts using the Bluesky API, but please note that this may not be the case for other social media platforms.
62
62
63
-
{:style="max-height:500px"}
63
+
{:style="max-height:500px"}
64
64
65
65
Before you begin to collect data, you will need to choose a topic and find your hashtags. To choose a hashtag or a simple keyword, please go the [the “Choose your hashtag” section](#section3-1) below. Once you know what you are looking for, click on "My Datasets" and choose the "Historical Posts" option under the Bluesky tab.
66
66
@@ -72,7 +72,7 @@ As you see, you will be asked to:
72
72
73
73
You will have to fill in a form to register all this information.
74
74
75
-
{:style="max-height:500px"}
75
+
{:style="max-height:500px"}
76
76
77
77
You can use a very simple search query or something more complex (see below for more details). You might want to test this out manually first in the Bluesky interface, which offers advanced search options [(Tips and Tricks for Bluesky Search)](https://bsky.social/about/blog/05-31-2024-search), to get an idea of the results it generates. The option you choose to harvest posts, whether by thread, timeline or keyword-based search, will depend on your research. The thread or timeline options might be useful, for instance, to study the interactions between a museum and its audience. You might want to limit your query to posts published by the museum’s accounts and the replies to these posts using the timeline option and its filters.
78
78
@@ -84,16 +84,16 @@ You should also consider the expected size of your corpus of posts. How big shou
84
84
85
85
Some important points to understand to help you maximise your query:
86
86
87
-
- You can create complex queries using keyword and hashtag combinations. The Bluesky API specifies certain search operators that work similarly to Boolean operators and are not case sensitive: you can search for exact matches by “putting statements inside double quotation marks”; if you want to search for posts containing all the words or phrases in your query, simply use a space between them, e.g. 1916 commemoration; the Boolean OR statement is acheieved by using a vertical line between words, e.g. commemoration | remembrance. For example, if I want to find posts about the commemoration of the women involved in the 1916 Easter rebellion in Ireland, “women centenary” would not be specific enough; a better search query would be women centenary #1916rising | “easter rising”. [See more about writing Bluesky search queries here](https://communalytic.org/docs/bluesky-historical-posts-data-collector/#step-2nbsp).
88
-
--Depending on the topic you are working on, it may not suffice to follow a small set of hashtags to build your corpus. Topics like #Trump or #ukrainewar produce a large number of posts a day; however, with some exceptions, commemorations generate far fewer posts. If your preferred topic is likely to produce a very small corpus which might hinder a meaningful analysis, you may need to use keywords in addition to hashtags. This type of query can become complex, but still you can test it using the Bluesky search bar to make sure it returns the expected results before implementing it in Communalytic. To create the dataset in Communalytic, click on the “Start Data Collection” button. A new page will then be loaded while the data is being captured.
87
+
- You can create complex queries using keyword and hashtag combinations. The Bluesky API specifies certain search operators that work similarly to Boolean operators and are not case sensitive: you can search for exact matches by “putting statements inside double quotation marks”; if you want to search for posts containing all the words or phrases in your query, simply use a space between them, e.g. 1916 commemoration; the Boolean OR statement is acheieved by using a vertical line between words, e.g. `commemoration | remembrance`. For example, if I want to find posts about the commemoration of the women involved in the 1916 Easter rebellion in Ireland, “women centenary” would not be specific enough; a better search query would be women centenary `#1916rising | “easter rising”`. [See more about writing Bluesky search queries here](https://communalytic.org/docs/bluesky-historical-posts-data-collector/#step-2nbsp).
88
+
- Depending on the topic you are working on, it may not suffice to follow a small set of hashtags to build your corpus. Topics like #Trump or #ukrainewar produce a large number of posts a day; however, with some exceptions, commemorations generate far fewer posts. If your preferred topic is likely to produce a very small corpus which might hinder a meaningful analysis, you may need to use keywords in addition to hashtags. This type of query can become complex, but still you can test it using the Bluesky search bar to make sure it returns the expected results before implementing it in Communalytic. To create the dataset in Communalytic, click on the “Start Data Collection” button. A new page will then be loaded while the data is being captured.
89
89
- You can build your corpus over a few days if necessary. You may, however, need to anticipate a major/annual commemoration or historic event. Some examples of annual “memorial” hashtags include: #WW1 #LestWeForget #Bloomsday #ArmisticeDay.
90
90
- If there are no posts that match your query, you will receive an error message.
91
91
92
-
{:style="max-height:500px"}
92
+
{:style="max-height:500px"}
93
93
94
94
Congratulations! You now have your first Bluesky dataset and can analyse it in various ways. A summary will appear at the bottom of the page. To begin analysing your data, click on the magnifying glass symbol under “Records” to access an overview of the dataset. From here, you can also download a spreadsheet containing the posts in Excel or CSV format if you wish to use another analysis tool. When you go to “Download Dataset”, you will also see a table of the posts that you have collected, with a search bar at the top if you want to search individual posts. You can export your data and analyse them with any method and tool that you prefer – we mention some at the end of the next section. We also strongly recommend that you download your dataset to preserve your data so that you can reuse them in your future research activities.
95
95
96
-
{:style="max-height:500px"}
96
+
{:style="max-height:500px"}
97
97
98
98
### Text analysis <aid="section3-2"></a>
99
99
@@ -103,29 +103,29 @@ Once you have your dataset, as shown in the [previous step](#section3-1),go to
103
103
104
104
Next, let’s see how you can explore topics within the dataset. The “Topic Analysis” function creates groups, or “clusters”, of posts that are similar in meaning to each other. These “cluster embeddings” may then allow you to identify abstract concepts, topics or themes within the dataset. Using the default clustering algorithm, click “Visualize Embeddings” [embedding = numerical representation of information] – this will be queued in the server and you may need to wait a few moments. Then click on “Open Visualization” to view and explore the word clusters, which will appear as data points in a “similarity map”. You can then adjust the clustering parameters and analyse the data again. For example, if you lower the “minimum cluster size” you will see a higher number of distinct clusters of posts. Use the exploration tool in the sidebar to zoom in on different clusters and label them. You may also download these data as a CSV file, if you wish – and again, we strongly recommend that you download and preserve any data you use for your analysis.
105
105
106
-
{:style="max-height:500px"}
106
+
{:style="max-height:500px"}
107
107
108
108
By now you have already learned a lot, so let’s take some time to reflect. Communalytic has allowed you to generate and explore a word cloud based on your corpus, but how exactly did this help you to understand and analyse your data? How useful has this been for your research, if at all? To help you think through these questions, you can read this article that discusses the limits of [word clouds](https://towardsdatascience.com/word-clouds-are-lame-263d9cbc49b7). You can even try to use other tools and compare the visualisations that you get: [Voyant Tools](https://voyant-tools.org/) is one option, but you can browse the [Text Analysis Portal for Research (TAPOR)](http://tapor.ca/home) to pick your preferred tool. Note that using different tools may require you to do some data cleaning - for instance, you might observe that Bluesky handles are appearing in the word cloud and this is not of great interest - or special formatting. [A detailed article on data cleaning can be found on Wikipedia](https://en.wikipedia.org/wiki/Data_cleansing). You can also discuss your experience with your colleagues or classmates or even a community of users.
109
109
110
110
### Network analysis <aid="section3-3"></a>
111
111
112
112
Network analysis is a second kind of analytical operation that can be applied to a social media data corpus. If you are not familiar with this field, [Marten Düring's lesson on data extraction and network visualisation of historical sources](https://programminghistorian.org/en/lessons/creating-network-diagrams-from-historical-sources) offers a nice introduction for beginners. [Once you have your dataset](#section3-1), click on the “Network Analyzer” module and you will see several options for different types of “ties”. Ties represent interactions between Bluesky accounts, which can take several forms: replying (using the “reply” functionality: your post will then appear below the original post and start or continue a conversation) and quoting (quoting another account’s post with a comment); reposting (quoting another account’s post with no modification); mentioning (publishing a post with another account’s handle in it); and link sharing, in other words “account-to-website” ties. The Bluesky user accounts are the nodes of the network you wish to inspect and the ties are the relationships that connect these nodes.
113
113
114
-
{:style="max-height:500px"}
114
+
{:style="max-height:500px"}
115
115
116
116
Choose the type of interaction you want to examine and click on “Generate Network”. You will have various options: you can download the network data as a GraphML file or a Gephi file, “Visualize Network” or choose a different type of interaction to work with.
117
117
118
118
Here there are two important things to mention. First, the nodes – the Bluesky accounts – are grouped in clusters which means these accounts share some form of similarity. These clusters are calculated by [an algorithm that detects how the Bluesky accounts are connected and figures out the best way to arrange these connections visually](https://rdrr.io/cran/igraph/man/layout_nicely.html). Second, you can [choose a layout](https://communalytic.org/docs/network-analyzer/#network-layout) to visualise the network you are analysing. A layout is a way to display the nodes (Bluesky accounts) and the connections between them (in graph theory, these connections are also called “edges”, which is another way of saying ties or links). Layouts are based on algorithms, which is why there are different possible layouts. A layout allows you to see patterns in the structure of your network. The best layout is the one that fits your dataset and your research purpose – there are no “good” or “bad” layouts per se.
119
119
120
-
{:style="max-height:500px"}
120
+
{:style="max-height:500px"}
121
121
122
122
You can use the Communalytic built-in visualiser to experiment with different layouts and see what kinds of visualisations they produce for your network. You can also take a closer look at what constitutes your network. For example, you can explore metrics that determine [the centrality of the network’s nodes](https://en.wikipedia.org/wiki/Centrality#Degree_centrality). Feel free to make these metrics visible by playing with the layout algorithms and exploring various filters. If you wish to focus on different parts of your network, you can explore “Community Clusters”. This will help you to isolate and focus on larger nodes, giving you a less detailed but clearer view of your network. Larger nodes represent Bluesky accounts with more interactions with other accounts, the highest being ranked first.
123
123
124
124
Communalytic also allows you to layer multiple types of analysis within the network, for example by running a “Sentiment” analysis on the text within the posts, which is to say identifying and describing "positive", "neutral" or "negative" emotions or opinions about a topic within a corpus of text. This uses [natural language processessing (NLP)](https://en.wikipedia.org/wiki/Natural_language_processing) to score posts for positivity, neutrality or negativity and can then be represented in the density of edges in the network visualisation. This may assist in thinking about the mentalités of user groups around an historical topic, or simply help you to contextualise themes found through text analysis. For a more detailed look at the ways in which sentiment analysis can be used to analyse social media data, see [this book chapter](https://link.springer.com/chapter/10.1007/978-3-031-52719-7_6). Likewise, the "Civility" analysis option (formerly the "Toxicity Analyzer) can help to identify posts containing either harmful or helpful ("prosocial") language. However, bear in mind that the interpretation of language in a social media dataset will also depend on your research questions and must be considered in context. All of the techniques outlined above are different ways of representing the data – but remember, visualisations are interpretations, they are not raw data. If you are happy with the results, you can also save and export your visualisation to integrate it into a future report or presentation.
125
125
126
126
How can network visualisations be useful for historians? There is a vast bibliography of publications that explore this question, but it is not the object of the current tutorial. What is useful to remember, however, is that you need to fix and focus on a research objective that is related to a historical question. You may benefit from [Martin Grandjean's general introduction to social networks analysis in history](http://www.martingrandjean.ch/introduction-to-social-network-analysis/),which will also give you insights into network metrics and structures such as degree centrality and communities. These are metrics that help analyse social media, specifically posts and networks, to identify dominant voices (nodes/Bluesky accounts) in the network (official, news media, themed accounts, cultural institutions, individual influencers, etc.).
127
127
128
-
{:style="max-height:500px"}
128
+
{:style="max-height:500px"}
129
129
130
130
As for text analysis, you can also download your network’s data and perform further analyses with [Gephi](https://gephi.org/) software or your preferred social network analysis tool. See [Martin Grandjean's tutorial](http://www.martingrandjean.ch/gephi-introduction/) to learn how to use Gephi.
0 commit comments