Getting Started

This project requires a running instance of Mastodon, a federated open source social media platform.

Mastodon developers offer a solid docker-compose option for ready to go configuration.

Therefore a cloud instance is required ex. Google Cloud, Amazon Web Services, Linode, DigitalOcean

We recommend an Ubuntu 20.04 LTS machine with 2 GB of RAM and min 20 GB of disk space.

Note

A domain name is also required for production. It can be acquired with GitHub Student Developer Pack. See further
.tech domains are recommended.

Install Docker

Second step is to install Docker Engine on Ubuntu.

Set up the repository

  1. Update the apt package index and install packages to allow apt to use a repository over HTTPS:

     sudo apt-get update

 sudo apt-get install \
 apt-transport-https \
 ca-certificates \
 curl \
 gnupg \
 lsb-release

  2. Add Docker’s official GPG key:

     curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

  3. Use the following command to set up the stable repository.

     echo \
 "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
 $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

  1. Update the apt package index, and install the latest version of Docker Engine and containerd

     sudo apt-get update
 sudo apt-get install docker-ce docker-ce-cli containerd.io

  2. Verify that Docker Engine is installed correctly by running the hello-world image

     sudo docker run hello-world

Install Docker Compose

Install Compose on Linux systems

  1. Run this command to download the current stable release of Docker Compose:

     sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

  2. Apply executable permissions to the binary

     sudo chmod +x /usr/local/bin/docker-compose

  3. Test the installation.

     docker-compose --version

If any problems with these steps read further: Install Docker Engine on Ubuntu

Install Docker Compose

Manage Docker as a non-root user

To create the docker group and add your user:

  1. Create the docker group.

     sudo groupadd docker

  2. Add your user to the docker group.

     sudo usermod -aG docker $USER

  3. Log out and log back in so that your group membership is re-evaluated.

    If testing on a virtual machine, it may be necessary to restart the virtual machine for changes to take effect.

    On Linux, you can also run the following command to activate the changes to groups:

     newgrp docker

  4. Verify that you can run docker commands without sudo

     docker run hello-world

Mastodon Docker Setup

Setting up

Launch a console terminal on the remote machine.

Clone Mastodon’s repository.

# Clone mastodon to ~/mastodon directory
git clone https://github.com/mastodon/mastodon.git mastodon
# Change directory to ~/mastodon
cd ~/mastodon
# Checkout to the latest stable branch
git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)

Installing Docker containers

Docker Images

You can use a prebuilt Docker image. Images are available from Docker Hub:
Mastodon Docker Image

Docker Compose and and environment files

Use these pre-configured files to avoid any problems. Github

Building the app

Copy the files .env.production and .env.db that have been provided to the /mastodon directory. Then change the docker-compose.yml file with provided one.

In env.production file replace LOCAL_DOMAIN with your own domain name.

To create an account for admin run the followings:

docker-compose run --rm web bundle exec rake mastodon:setup

This is an interactive wizard that will guide you through the basic and necessary options and generate new app secrets.

App secrets part did not worked for me before so we generated them beforehand.

  1. Enter the Fully Qualified Domain Name (FQDN) of your mastodon instance.

  2. Select if you want a Single User instance (not recommended, but if you prefer, use that).

  3. Obviously, you are running mastodon in a docker instance, so type Y (or hit return, as it’s the default)

  4. The PostgreSQL host is db, the port is 5432 (again, default), the database is mastodon_production, the database user is mastodon and the password is the one from docker-compose.yml.

  5. The redis server is redis, the port is 6379 and the password is empty.

  6. If you want to store uploaded files on the cloud, enter Y here – I haven’t tested that, but I expect you need an S3 or other cloud storage for that.

  7. If you want to send emails from the local machine, enter Y – considering we’re in a docker environment, I have only used my local STMP server in FQDN form, not localhost. Enter port, user and password for SMTP submission. Select the SMTP authentication type (when submitting locally, plain should be fine). Decide if you want to verify the identity of the server and, if so, what type of verification you want to do. Choose what sender address the emails will have (I use mastodon@*my.domain*).

DNS Configuration

Add A Record to instance IP with your DNS manager.

Reverse Proxy

You need a Reverse Proxy in front of your Mastodon instance. The preferred software for this Caddy 2 which is already a part of docker-compose.yml file that provided.

Caddy 2 is still the only web server to use TLS automatically and by default. Caddy 2 enables to deploy and scale HTTPS effortlessly.

Caddyfile Configuration

You need to configure Caddy to serve your Mastodon instance.

Reminder: Replace all occurrences of example.com with your own instance’s domain or sub-domain.

Create a Caddyfile containing only these lines and replace your.domain.tech with your own domain:

    your.domain.tech {
    reverse_proxy web:3000
    }

Launch Mastodon

After it’s done, you can launch Mastodon with:

docker-compose up -d

Within ~30 seconds all docker containers should be up and running. You can check it with docker ps command.

Creating accounts

Main admin account is created when building the app. Any other accounts must be created and confirmed on the terminal with admin CLI that Mastodon provides. Since an e-mail delivery service or other SMTP server is not dedicated for the server, its only possible to confirm users with the admin CLI tool.

Execute the following command on the terminal where the docker-compose.yml file exists to create an account

    docker-compose run --rm web bin/tootctl accounts create USERNAME --email EMAIL --confirmed

Replace USERNAME and EMAIL with relevant information.

If an account is created at sign-up page and waiting for confirmation, user can be modified with the following command:

    docker-compose run --rm web bin/tootctl accounts modify USERNAME --confirm

For further information about admin CLI, you can check out Using the admin CLI

Dashboard and API

Clone this project’s repository.

git clone https://github.com/dogukanburda/Developing-interfaces-for-social-network-experiments.git 
# Change directory to ./Developing-interfaces-for-social-network-experiments
cd Developing-interfaces-for-social-network-experiments

Local development

Install dependencies

To install all dependencies please run requirements.txt file first:

pip install -r requirements.txt

Preview the project

To view dashboard on your local host run:

streamlit run main.py

Deployment

Dockerize the Streamlit web-app

Build the docker image from the Dockerfile in the current folder

docker build -t streamlit-dashboard .

Run the built image with

docker run -p 8501:8501 streamlit-dashboard:latest

Project Components

Steamlit

Streamlit is an open-source app framework used to create a Dashboard with Python.
main.py is the main frontend source file to run the project.

Pure.py

  • serves as a server communicator between Mastadon and Dashboard
    Built on top of the Mastodon.py python wrapper. Further information:

Mastodon.py

Creating an actual Mastodon API instance:

By creating a Pure object you create an actual Mastodon API instance. It has an API that allows you to interact with any given Mastodon server’s every aspect.
Generate your own secret files with provided methods and substitute your secret files to the Pure object.

Register

Register your app! This only needs to be done once. Uncomment the code and substitute in your information:

from pure import Pure

'''
Pure.create_app(
     'PURE',
     api_base_url = 'https://mastodon.social',
     to_file = 'app_cred.secret'
)
'''

Login

Then login. This can be done every time, or you can use the persisted information:

from pure import Pure

pure = Pure(
    client_id = 'app_cred.secret',
    api_base_url = 'https://mastodon.social'
)
pure.log_in(
    'my_login_email@example.com',
    'incrediblygoodpassword',
    to_file = 'user_cred.secret'
)

To post, create an actual API instance:

from pure import Pure

pure = Pure(
    access_token = 'user_cred.secret',
    api_base_url = 'https://mastodon.social'
)
pure.toot('Tooting from python !')

Pure object’s functionality :

  • Get a title and a description from the server - when conducting research on other mastodon servers your dashboard will automatically update its title

  • Get a timeline of specific logged in user

  • Create a dataframe which includes {user_ids, usernames, toot_ids, toot_time, favourites_count, toot_content}

  • Get user_id from a provided username

  • Provide user_toots from a given user_id

  • Create a Followings Network of 400 users starting from given a user_id

  • Create detailed .html file that includes account_information from a provided username

  • Get clean content of last 400 toots from a provided user_id which is implemented to create a Word Cloud

Main.py

Main.py functionality:

  • Implement a Pure object

  • Display Options Menu : User Toots, User Profile, Diffusion Netwok, Word Cloud

  • If user selects User Toots:

    • Takes username as input and calls get_user_id to communicate with Mastodon API and calls get_user_toots method of Pure object

    • Plots a graph of toots’ favourites_count versus toot_time

    • Graph can also display details about each toot when hover on it

  • If user selects User Profile:

    • Program displays .html file that includes information about user

    • Display Account Statistics: - Toots by day of the week - Toots timeseries

  • If user selects Diffusion Netwok:

    • Takes username as an input and displays a Diffusion Network graph.

  • If user selects Word Cloud:

    • Program gets toots’ content from Pure object, and creates a word cloud in shape of a mastodon

Bot Template

  • Listener:

    • Listener class inherits mastodon.py’s StreamListener class.

    • Listener class listens the server with its every method and executes the followings according to corresponding server response.

    • By default follows back when a user follows the logged in account.

    • Replies mentions with Dont disturb me 😡. sentence.

    • Replies the toots on the current stream with sentiment analysis scores.

# Main object listens to the stream
class Listener(StreamListener):
    def __init__(self,bot,pure):
        # self.bot for handling content
        self.bot = bot
        #self.pure for taking actions on the server with given credentials
        self.pure = pure

    def on_notification(self, ntf):
        # Follows back whoever follows the logged in user
        if ntf['type'] == 'follow':
            print("You have a new follower !",ntf['account']['username'])
            self.pure.account_follow(ntf['account']['id'])
            print("You followed back {}.".format(ntf['account']['username']))
        #print(ntf)
        # Replies mentions with 'Dont disturb'
        elif ntf['type'] == 'mention':
            print('Someone mentioned you !',ntf['account']['username'])
            self.pure.status_post('Dont disturb me 😡.',in_reply_to_id=ntf['status']['id'])

    def on_update(self, status):
        """A new status has appeared! 'status' is the parsed JSON dictionary
        describing the status."""
        # Detects language of the status and does a sentiment analysis with transformers pipeline
        toot_id = status['id']
        lang = self.bot.toot_language(status)
        label_score = self.bot.toot_sentiment(status)
        string = "This status' sentiment analysis is classified as {}, with confidence of {}. Model provided from transformers 🤗 (https://github.com/huggingface/transformers)".format(label_score[0]['label'],label_score[0]['score'])
        pure.status_post(string,in_reply_to_id=toot_id)
        context = self.bot.context(status)
        print('Context: ', context)
        print('Detected language: ', lang)

    def handle_heartbeat(self):
        """The server has sent us a keep-alive message. This callback may be
        useful to carry out periodic housekeeping tasks, or just to confirm
        that the connection is still open."""

        print('Bot script is working...')

  • MastodonBot:

    • Bot class for handling content.

      • Cleans the returned values’ content for analysis.

      • Detects the language and sentiment of shared content.

    • This can be customized for the user’s needs.

# bot features
class MastodonBot():
    def __init__(self):
        pass

    def cleanhtml(self,raw_html):
        import re
        cleanr = re.compile('<.*?>')
        cleantext = re.sub(cleanr, '', raw_html)
        return cleantext

    def toot_language(self,status):
        from textblob.sentiments import NaiveBayesAnalyzer
        from textblob import TextBlob
        global lang_dict
        toot_context = self.cleanhtml(status['content'])
        blob = TextBlob(toot_context,analyzer=NaiveBayesAnalyzer())
        detected_language = blob.detect_language()
        detected_language_name = lang_dict[detected_language]
        return detected_language_name

    def toot_sentiment(self,status):
        from transformers import pipeline
        classifier = pipeline('sentiment-analysis')
        label_score = classifier(self.cleanhtml(status['content']))
        return label_score

    def context(self,status):
        toot_context = self.cleanhtml(status['content'])
        return toot_context

Running the bot

  • First create a MastodonBot object for handling content

  • Create a Pure object with your credentials to communicate with server.

  • Initialize the Listener object with the MastodonBot and Pure objects.

  • Connect the Listener object to desired stream.

    • For streaming options, check Streaming link.

Example bot initalization.

bot = MastodonBot()
pure = Pure(access_token='./secrets/pure_user.secret',api_base_url='https://mastodon.social')
listener = Listener(bot,pure)
pure.stream_local(listener)