Twitterbots - Overview

Updated January 2021

Content warning: page contains a potty-mouthed feline

Lets talk about Twitter...

No don't go, please stay on my site, please

Does anyone use it any more?

What do you with your account when you've got bored of it? Archive it? Delete it?

You could always automate it with the help of a Raspberry Pi?

At the moment, I have four automated Twitter accounts, they are The Ford Fiesta, Vader the Cat, Ian Hill, and Arse Clock, all running from one headless Raspberry Pi 3B (which is buried in a drawer near my router with another Pi 3B, a 3B+, a dedicated NAS, and a gigabit switch) and all four of them are completely automated, including random tweets, new follower messages, and replying to mentions while saving everything it does to a log file which can be viewed via it's web page, the web page allows some basic control of the engine including sending tweets on demand



Pages in this series...

Page 1: Overview
Page 2: Tweeting randomly
Page 3: Tweeting pictures
Page 4: Tweeting new followers
Page 5: Tweeting replies to mentions
Page 6: Scanning for trending hashtags
Page 7: Automating everything


<pause for="rant">
Once in a while either myself or "Vader" will get somebody complaining this is a fake automated account, either verbally, or via Twitter itself, and I'm posting the tweets myself. It makes no difference that Vader posts once, up to every three hours (or 10800 seconds) all day, every day, and the fact new followers are scanned for once every three hours, all day, every day, finally the mentions are scanned once an hour, on the hour, all day, every day.
None of this make any difference whatsoever, so what better way to prove it is a real fully automated account, than to explain how it works and provide some of the source code...
</pause>

And continue...

My Twitterbot history

This is where it all started, as one of those random ideas I had during the #projectphone experiment



Yes, it did annoy people, a lot 🤣

Project Phone controlled Twitter by way of text messages, Twitter knew the phone number associated with the account, and post tweets on the timeline when texts from that number were received, and if anyone replied, or followed the account, a text message would be sent to the phone, which the Project Phone software would read, analyse, and respond to, all controlled by a scripting language called Mortscript, which controlled other apps installed on the system, it was a bit of a mess, but it did work

These days you need to create an app at Twitter, you will need one app for each account you want to automate, however it's possible to control them all from one script on the Pi, if you can get your head round it all!

We'll take a look at Vader's account, as it's a good place to start, has the most features, and "coincidentally" the most popular...

Caution! Vader is a real cat, AKA the world's angriest cat, and he tweets his innermost thoughts, usually about taking over the world, and the destruction of the human race, this means his language can be appalling sometimes



Today, my Twitterbots are a lot more sophisticated, Vader's Twitterbot has a total bank of 448 tweets, and depending on factors like the time of day, or day of the year etc, some of the tweets are pulled into another array where one gets chosen at random and tweeted, this of course can include variables, so Vader can tweet what the time is, the current day, pictures, or ask questions about whatever hashtag is trending at the time



The Python script can also scan for the top 20 trending hashtags and use them in it's tweets



Additionally there are scripts that check for new followers and any mentions that may have happened recently




Also, the script can tweet the occasional photo...



And some more random tweets...



You may notice the text VaderTheCat at the bottom of some of the tweets, this is the name of the Twitter app I wrote to control the account, if the tweet was posted by the official Twitter app, it would say Twitter for Android/iOS etc. This shows it's a third party app controlling the account, whether it's automated or not

Here are some screen shots of the web portal running on the Pi



You can see that it gives information about the Pi itself, and you can't do anything unless you log in. Once you've logged in, you can access each Twitterbot's control panel, perform a backup or reboot the server etc, and it shows if a script is running in the background

The control panel for each Twitterbot allows you to post a tweet or a picture on demand, see when the last tweet was sent, or when next tweet will be posted. You can also access the most recently saved followers or mentions lists, access the log files. Finally a recent addition was the ability to upload images to be tweeted to the system

If this sounds like something you want to do, then read on...


Create an app at Twitter

Firstly, you need to apply for a developer account at Twitter, so go to twitter on a PC, and log in to your account, then go to the Twitter Developers page and apply for a developer account (I never had to do this when setting up my apps, so it's new to me too)

Selecting Hobbyist and Making a bot seems the most appropriate, you must give a valid email and phone number (which must be verified) when setting up your account, and you can use the same number for multiple apps

You will need to describe exactly what you will use the app for, so be as honest as you can. Once complete you will need to confirm your email address, and wait for their decision...

Yes, it can take several days to receive the decision...

Once you get your account set up, go back to the Twitter Developers page and click on the Apps button in the top right, then Create an app

Fill out the form as best as you can, you will need to describe your intentions with the app, so be honest as you can, there is a minimum character count, so put as much as you can.

Don't enable sign in with Twitter

You'll need a website URL, I just put ianwilliamhill.co.uk

Once done, click Create

Read the terms and conditions reminder



Go to the permissions tab and set the access permission to read, write, and direct messages, then go back to the keys and tokens tab and regenerate the consumer API keys, then create the access token and access token secret keys.



Copy all four of these keys down somewhere now, if you navigate away from this page, Twitter may not let you access them again, and you'll have to regenerate them

That's because a recently added new and annoying feature is that you need to regenerate the keys if you make any changes to the app, and you only get once chance to view/save the keys, then you'll need to regenerate the keys again, so make sure everything is set up before you generate the keys, and you save them somewhere before closing the tab

Once you've gone through all the above to get the four access keys, replace the abcde... with your unique keys in the code below, once this is complete, you are finished with the Twitter Developer page, and it can be closed

It may be a good time to read the developer terms and conditions, because it's easy to get these keys revoked and your app blacklisted (just create a new app and get new keys if you do), normally by sending unsolicited tweets to other users. It's also it's a good idea to learn about Twitter's rate limiting policy, this is to prevent you tweeting too much and spamming the platform, if you get rate limited, you will be locked out from accessing the Twitter API for a while, so it's not a good idea to keep running the scripts while testing your code



Setting up the Pi

Get hold of a spare Pi (you should have one) any will do, the Twitterbot code will happily run on the original model B with 256MB of RAM.
Write the latest Raspbian lite to a 16GB or larger SD card (Buster at the time of publishing), I would recommend using an Ethernet cable over Wifi for network reliability, this Pi will run headless and can be controlled via a web page on the network

I would recommend the tools WinSCP, putty, and Notepad++ to work on the Pi

Update May 2020

Due to a recent SD card failure I had to start over and found Twython has been updated to work with Python 3 which breaks compatibility with Python 2, you can install it under Python 3 but the scripts here don't run under Python 3 very well, so the install instructions have been updated below

The latest version (at time of publishing) is 3.8.0, this will not install under Python 2 on Raspbian Buster

So I recommend using Raspbian Stretch and Twython 3.7.0 (Download from the zipball link) This version of Twython may install on Raspbian Buster and later versions, but I didn't try it

Install Stretch as normal, and update using...

pi@raspberrypi:/$ sudo su
root@raspberrypi:/# apt-get update && apt-get upgrade -y

Then on your PC extract the file ryanmcgrath-twython-3.7.0-29-g02fb356.tar and transfer the extracted folder to the Pi, putting it in /home/pi will do

On the Pi navigate through the extracted folder(s) until you see the file setup.py, then use the commands

pi@raspberrypi:/$ sudo su
root@raspberrypi:/# python setup.py install
root@raspberrypi:/# apt-get install apache2 php

Update January 2021

If you are using Raspbian Buster or Raspberry Pi OS the the above process may fail with a "TypeError: 'encoding' is an invalid keyword argument for this function".

To fix this you need to open setup.py in Notepad++ and add the line

from io import open

to the start of the file just after the line

import sys


Depreciated instructions

Skip to Setting up a new Notepad++ file below. Do not use these commands

Once that's all set up and you've logged in to the Pi, either by SSH or on the Pi itself type...

pi@raspberrypi:/$ sudo su
root@raspberrypi:/# apt-get update && apt-get upgrade -y
root@raspberrypi:/# apt-get install python-pip
root@raspberrypi:/# pip install twython
root@raspberrypi:/# apt-get install apache2 php

Some of these packages may be installed already


Setting up a new Notepad++ file

Open Notepad++ and create a new file, then go to Edit > EOL Conversion > Unix (LF) to set the text file to Unix format so the Python interpreter on the Pi can read the file, if you don't do this, you'll get various invalid character errors




Enter the code samples below (but with your keys) and save the file, then transfer it to the Pi in /home/pi using WinSCP

Python is fussy about the indentations, I normally use three spaces inside conditional statements and loops, but others may use two or four spaces, if you copy code from elsewhere, make sure they all match

You can either ssh into your Pi with Putty to run the scripts, or use an attached USB keyboard and do it on the Pi itself

Python code for text tweets

To send a tweet from the Pi is quite easy and doesn't require much code

Copy and paste the following in

#!/usr/bin/env python
import sys
import os
import subprocess
import time
import datetime
from twython import Twython
from time import strftime

CONSUMER_KEY = 'YOUR CONSUMER KEY GOES HERE'
CONSUMER_SECRET = 'YOUR CONSUMER KEY SECRET GOES HERE'
ACCESS_KEY = 'YOUR ACCESS KEY GOES HERE'
ACCESS_SECRET = 'YOUR ACCESS SECRET GOES HERE'

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

try:
   twitter.update_status(status="Hello world :)")
except:
   print str(sys.exc_info())

You will need to enter your key details between the two quotes ' '

Save this file as updatetwitter.py and transfer it to the Pi

Now run the file

pi@raspberrypi:/$ python updatetwitter.py

All being well, after a few seconds, your Twitter account will have Tweeted "Hello world :)"

If it has, then it's all good...


Python code for picture tweets

Now we can try to tweet a picture, so create a folder in /home/pi called images

pi@raspberrypi:/$ mkdir /home/pi/images

Put any image in the folder and rename it "picture.jpg", then go back to Notepad++ and create another new file with Unix line terminators as above and paste the following code...

#!/usr/bin/env python
import sys
import os
import subprocess
import time
import datetime
from twython import Twython
from time import strftime

CONSUMER_KEY = 'YOUR CONSUMER KEY GOES HERE'
CONSUMER_SECRET = 'YOUR CONSUMER KEY SECRET GOES HERE'
ACCESS_KEY = 'YOUR ACCESS KEY GOES HERE'
ACCESS_SECRET = 'YOUR ACCESS SECRET GOES HERE'

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

photo = open("images/picture.jpg", 'rb')

image = twitter.upload_media(media=photo)

try:
   twitter.update_status(status="Hello world :)", media_ids=[image['media_id']])
except:
   print str(sys.exc_info())
   sys.exit()

Save this file as updatetwitterpicture.py and transfer it to the Pi and run it

pi@raspberrypi:/$ python updatetwitterpicture.py

All being well your Twitter account will have tweeted the picture with the caption "Hello world :)"

If it has, fantastic...



Getting new followers

Create a new file as before and call it followers.py, the @catdarthvader string in the code below can be from any account, this script will work for any account where the followers can be viewed

#!/usr/bin/env python
import sys
import os
import subprocess
import time
import datetime
from twython import Twython
from time import strftime

CONSUMER_KEY = 'YOUR CONSUMER KEY GOES HERE'
CONSUMER_SECRET = 'YOUR CONSUMER KEY SECRET GOES HERE'
ACCESS_KEY = 'YOUR ACCESS KEY GOES HERE'
ACCESS_SECRET = 'YOUR ACCESS SECRET GOES HERE'

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

username = "@catdarthvader"
delay = 120

next_cursor = -1

followers = []
while(next_cursor):
   try:
      get_followers = twitter.get_followers_list (screen_name=username, count=200, cursor=next_cursor)
   except:
      print str(sys.exc_info())
      sys.exit()
   for follower in get_followers["users"]:
      followers.append (follower["screen_name"].encode ("utf-8")+"\n")
      next_cursor = get_followers["next_cursor"]
   time.sleep(delay)

for a in followers:
   print a

If you run this file it will print all the followers out, if you look at the code, you can see the delay added into the loop, this is to avoid exceeding the rate limit set by Twitter, so if you have a lot of followers, it can take a long time. Vader has about 1500 followers, and it takes about 15 minutes to get them all and print them on the screen

pi@raspberrypi:/$ python followers.py

Also, you won't get told if any of the followers are new followers, you will need to store the list of followers somewhere, so the next time you scan for followers, you can compare the old list with the new list, and any followers in the new list which are not in the old list are new, or reactivated accounts, you will get followers that either hide their accounts, or get temporarily banned, when they are reactivated, they will be treated as a new follower. Vader sees the same few accounts getting new follower tweets almost daily, maybe I should add some code to detect this..?


Getting recent mentions

Create a new file as above and call it mentions.py

#!/usr/bin/env python
import sys
import os
import subprocess
import time
import datetime
from twython import Twython
from time import strftime

CONSUMER_KEY = 'YOUR CONSUMER KEY GOES HERE'
CONSUMER_SECRET = 'YOUR CONSUMER KEY SECRET GOES HERE'
ACCESS_KEY = 'YOUR ACCESS KEY GOES HERE'
ACCESS_SECRET = 'YOUR ACCESS SECRET GOES HERE'

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

mentions = []
users = []
twitter_id = []
tweetID = []
ts = int(time.time())

try:
   tweets = twitter.get_mentions_timeline(count=100)
   for tweet in tweets:
      users.append("@" + str(tweet['user']['screen_name']) + "\n")
      tweetID.append(str(tweet['id_str']) + "\n")
      s = str(tweet['created_at']) # Thu Oct 26 13:11:02 +0000 2017
      t = time.mktime(datetime.datetime.strptime(s, "%a %b %d %H:%M:%S +0000 %Y").timetuple())
      ti = int(t)
      mentions.append(str(ti) + " # @" + tweet['user']['screen_name'] + " - " + tweet['text'].replace('\n', ' ') + " - " + tweet['id_str'] + "\n")
      tp = ts - ti
      twitter_id.append(tweet['id_str'])
except:
   print str(sys.exc_info())
   sys.exit()

for tweet in mentions:
   print tweet

In the mentions script, you can see I get the last 100 mentions, this is enough if you run the script once an hour, you can go up to the last 200 mentions, but if you want more, you will need to pull them down at 200 at a time two minutes apart, just like with the followers script. But just like with the followers, Twitter doesn't tell you if they are new mentions or not, so you will need to store the list and compare it with the next list you get.

Also the warning above about reactivated accounts applies here too

pi@raspberrypi:/$ python mentions.py

You may also notice this script formats the mentions so each one gives you the important information without all the bloat that usually comes with each mention block, so they can be processed easier by the script, they are formatted like this...

{useraccountnumber} # {@username} - {tweetbody} - {tweetid}


Getting trending hashtags

I added the code in white text into the updatetwitter.py script, so Vader can occasionally ask his followers about trending hashtags as one of his many random tweets, but I made it it's own script here, you can call it trends.py

#!/usr/bin/env python
import sys
import os
import subprocess
import time
import datetime
from twython import Twython
from time import strftime

CONSUMER_KEY = 'YOUR CONSUMER KEY GOES HERE'
CONSUMER_SECRET = 'YOUR CONSUMER KEY SECRET GOES HERE'
ACCESS_KEY = 'YOUR ACCESS KEY GOES HERE'
ACCESS_SECRET = 'YOUR ACCESS SECRET GOES HERE'

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

try:
   trends = str(twitter.get_place_trends(id=23424975))
   newtrend = trends
   newtrendlength = len(newtrend)

   a=[]
   b=[]
   aa = 0
   for i in newtrend:
      if i == "#":
         a.append(aa)
      aa = aa + 1

   for f in a:
      s = newtrend[f:newtrendlength]
      t = s.find("'")
      b.append(s[:t])
except:
   print str(sys.exc_info())
   sys.exit()

for tweet in b:
   print tweet

The number 23424975 is for the UK, there are ID numbers for all the countries in the world, or it does go down to city level if that's what you prefer. You can look them up by using the Yahoo Where On Earth Identifier database (WOEID), or you could if you can find a working one!. You will also find that once you've accessed the trending hashtags, they are cached for a while, so if you try to access them again, you'll get the same list, and you'll nudge a little closer to your rate limit

pi@raspberrypi:/$ python trends.py

This is it for this tutorial

Check out part two where we start adding more code to generate random tweets...

For more project ideas check out my other projects here