Get Tweets From Multiple Tweeters
In this post I'll cover using the twitter-lite package to loop through an array of twitter users and fetch their tweets. You'll need a twitter developer account and app to get your set of keys to access the twitter api. This might take a day or two for them to verify your application, I suggest selecting the "Exploring The API" or "Student" options.
It will help to be aware of async/await and promises since they are very useful for making asynchronous actions like network api calls, querying a database and reading/writing to files. If you are unfamiliar with async/await then this is a great way to learn and see a practical use case of it in action. I sure learned about it putting the app together that this example is based on.
Twitter API
The Twitter API has many endpoints for getting all kinds of data, we'll be focusing on the statuses/user_timeline to get a set of recent tweets by providing a user_name or user_id. This will allow us to get other users' tweets which is kinda of creepy if you think about it. Luckily we'll be retrieving user's who are leaders and mover and shakers in the front end development community, so it should be a lot of cool stuff relevant to what we are doing in this tutorial.
The order of code operations will be this: 1) read a file of twitter users, 2) loop through these users and call a function to fetch their tweets, 3) fetch the tweets from the api, 4) aggregate all those tweets into one array, 5) do something like save to a database or file to eventually display somehwhere.
But first, you'll need your twitter app's keys and tokens which you can find in your developer account dashboard. We'll need to initialize our twitter API object with the required authentication credentials. I'm saving mine in environment variables, but you can easily replace them with the raw string. This first step is clearly spelled out in the twitter-lite documentation.
const Twitter = require('twitter-lite');
const twitter = new Twitter({
subdomain: "api",
consumer_key: process.env.CONSUMER_KEY, // from Twitter.
consumer_secret: process.env.CONSUMER_SECRET, // from Twitter.
access_token_key: process.env.ACCESS_TOKEN_KEY, // from your User (oauth_token)
access_token_secret: process.env.ACCESS_TOKEN_SECRET // from your User (oauth_token_secret)
});
Tweeters
I have collected a small list of front end tweeters to use for our query. I have stored these in the tweeters.json file. When we call the twitter api endpoint at statuses/user_timline, we will be using the twitter handle for each user and then get response with a bunch of fields. Scroll down to the bottom of the twitter api docs to see the data structure returned.
Our first function will read the tweeters.json file and parse the JSON to be used by another function. We are wrapping this in a promise and then putting the async keyword in front of the getTweeters() function to tell our code that this function returns a promise.Check out javascript.info for some great examples of how to use async/await. The reason we are using promises is to make our code asynchronous in order to handle tasks that take an uncertain amount of time to complete, hence it's called a promise for something in the future to occur.
The third line uses Node's file system library to readFile() which is our tweeters.json. The . in the file path says to look in the current directory, unlike a double .. which says to move up a directory. We use the promise callback keyword resolve to return our parsed json contents from the file. The reject callback will return the error if one occurs.
async function getTweeters() {
return new Promise((resolve, reject) => {
fs.readFile('./tweeters.json', (err, data) => {
if (err) reject(err);
else resolve(JSON.parse(data));
});
});
};
Getting The Tweets
Getting the tweets from the tweeters uses two functions, each function does a specific action to keep things nice and isolated.
The first function getTweets()loops through the array of user objects from our file, and calls another function getUserTweets() which actually makes the twitter api call. This was a crucial learning lesson for me on looping with async/await. You'll notice we are using a for loop and not something like forEach on our users array. Zell Liewexplains that forEach is not promise aware. Thus we have to use a for loop and it works just fine but definitely a subtle detailed gotcha to know if you try writing this code with forEach.
As we loop through the users, we collect their tweets in the allTweets array and then do a deep flatten on it. Each iteration returns an array of objects — [[{},{}],[{},{}]]. We want a nice flat array that this has not nested arrays — [{},{}].
async function getTweets(users) {
let allTweets = [];
for (let index = 0; index < users.length; index++) {
let user_tweet_list = await getUserTweets(users[index]);
allTweets.push(user_tweet_list);
}
return flattenDeep(allTweets);
};
The getUserTweets() will use the twitter object we created at the top of the file and call the get method and passing in the statuses/user_timeline endpoint along with the user.handle and a the number of tweets we would like to return. Once again, we are wrapping this code with a promise and resolving the userTweets or rejecting with the error. Take some time to read each line of code in the function, there is a lot of good stuff happening.
The if state checks to make sure the tweet is not a retweet of someone else's tweet, we only want original tweets from the user. It would help to see the data structure returned from twitter for a single tweet. After we get the data, we loop through it and save it to an object literal that will be added to the userTweets array.
async function getUserTweets(user) {
return new Promise((resolve, reject) => {
let userTweets = [];
twitter
.get("statuses/user_timeline", {screen_name: user.handle, count: 3})
.then(data => {
data.forEach(function(tweet) {
if (!tweet.retweeted_status) {
userTweets.push({
"twittertime": tweet.created_at,
"created": new Date(tweet.created_at),
"handle": tweet.user.screen_name,
"message": tweet.text,
"site": tweet.user.url || '',
"topic": user.topic,
"hashtags": [...tweet.entities.hashtags],
"link": `https://www.twitter.com/${tweet.user.screen_name}/status/${tweet.id_str}`,
"id_string": tweet.id_str
});
}
});
resolve(userTweets);
})
.catch(error => {
reject(error);
});
});
};
A Helpful Tool
HTTP Toolkit was posted on hackernews one day and I found it to be useful for api debugging and testing. I simply selected a terminal session which opened up a terminal window and I navigated to my app directory, ran the node script and you'll get the HTTP response and JSON from the Twitter API. Very helpful!
HTTP Toolkit
Saving the Tweets
Finally, we'll save the tweets to a file. You could easily save to a database as well.
async function saveTweets(tweets) {
fs.writeFileSync(`./tweets.json`, JSON.stringify(tweets));
};
Putting It Together
At last, we call all of our functions but instead of using promises to write code like below:
getTweeters()
.then(getTweets)
.then(saveTweets)
We can write the code that reads like synchronous code with async/await. It definitly is a much smoother way to write promises. Probably better for your eyes too.
async function getAllTweets() {
let tweeters = await getTweeters();
let tweets = await getTweets(tweeters);
await saveTweets(tweets);
}
getAllTweets();
This post was based on Front End Tweets, a project that fetches tweets from popular UI/UX/Front Ender's.