Tutorials

The Australian Internet Observatory Research Dashboard (AIOReD) is an interactive web based application designed to provide user friendly access to the API, to perform social media analysis. The dashboard allows users to interact with data present in the databases, along with some of the topic modelling results and word embeddings that are generated by the ‘live’ machine learning pipelines. The dashboard works best a stable internet connection, and on a laptop or desktop computer, as it can be network intensive...

The aggregation tab in AIReD provides ability to aggregate over the social media collections in four different ways, which called different endpoints at the API. Location Time (by day) Seasonality Language Each aggregation is controlled by a start and an end date, to specify the period over which the aggregation is desired. Upon loading the aggregation tab, one can select the start and end date by using the widget in the top left corner. Once these value are selected, all...

The AIReD topic graph tab provides users with an interactive way to examine the outputs of the Topic Modelling pipeline. The pipeline is run every day, on a days worth of social media data, using BERTopic, to generate a list of topic clusters. These clusters simulate the trending topics of the given day, and each cluster is associated with 30 representative terms – from which the topic of the cluster can be extrapolated. Topic graph construction In order to track the...

Two outputs of the Topic Modelling pipeline (which is run daily, on a days worth of social media data) are the frequency of terms, as well as the development of word embeddings for the day. The API grants the ability to analyze the term frequency data, as well as interact with the word embedding models. The AIReD Term analysis tab interacts with the API to visualize the outputs, and examine the semantic relationships between words in a certain time period...

Some steps in this tutorial require Python, versions 3.8 and greater, to be installed along with the python packages requests, matplotlib and wordcloud. The Twitter API contains an endpoint which allows authorized users (details on how to attain access is listed here) to attain variety of information about the Tweet specified by the requested tweet ID or list of tweet IDs. There are a number of ways in which an interesting list of tweet ids may be attained, one of...

The full-text search functionality allows selecting social media posts (for the time being only for the Twitter collection) based on some post attributes: The attributes can be either equal or not equal to a value entered by the user, forming a `’rule’. Rules can be composed in groups joined by logical operators (‘and’, ‘or’, ‘not’). The expression can be changed manually by editing it before pressing the ‘Execute query’ button. A ‘Reset’ button allows for clearing the expression. The results...

AIReD is harvesting Mastodon social media posts, and the social media post IDs can be downloaded from either the API or the Dashboard (for instance by executing text search queries). Once a text search is executed, some Mastodon post IDs can be downloaded in CSV format, for instance: The next logical step is to get the contents of said social media posts, which requires a bit of coding. To get the original social media posts, you first need to get...

To hydrate Reddit posts from the output of the AIReD API, this Jupyter notebook can be used. For simpler use case, this Python script, that does not require a Reddit API key, can be used (substitute the AIReD results in the URLS array):...