Read hashtags from replies

[cuducos]

This morning in a Twitter thread a @RosieDaSerenata follower gave us a great idea:

This bot could tweet (as it does today) but also read the replies @RosieDaSerenata gets with a couple of standardized hashtags (eg #FalsoPositivo and #RosieAcertaOutraVez).

This could be useful to build a dataset for supervised machine learning from the feedback we get from @RosieDaSerenata followers.

[cuducos]

Adding the proper credits for the idea: thanks @arcanj0 ; )

[arcanj0]

Very nice!

Would be very helpful for the toll itself, like you said.
The cons would be you could be exposed to Bots using the hashtag in a "Evil way", but I guess it's not that hard to identify!

The way it is today doesn't give the followers the certainty to be counted as a help.

I would love to see it working!

Thanks @cuducos !

[vmesel]

@cuducos, you are saying that you would like to create a dataset with the data:

recipe_id | count_positive | count_negative | real_classification

If you do, there is no API to checkout the replies for tweets, but there is a way you can scrape a user profile in order to find a reply_to_tweet method with the original @RosieDoSerenata tweet ID. The link to do this is: https://stackoverflow.com/questions/2693553/replies-to-a-particular-tweet-twitter-api

[cuducos]

First of all:

If you do, there is no API to checkout the replies for tweets

Ow, c'mon @vmesel — you're better than that ; )

Do you prefer to trust in a 2010 Stack Overflow reply or in Twitter API official documentation?

BTW there's a Twitter API Python wrapper I've contributed to that offers GetReplies method to list replies to a certain user.

The main question is: given a @RosieDaSerenata list of replies, are we able to identify which tweet is each reply is actually refers to using this wrapper? Twitter API offers in_reply_to_… properties.

That said we have to choose a API wrapper or to deal directly with the REST API.

create a dataset with the data:
recipe_id | count_positive | count_negative | real_classification

I'd prefer to have count as something like a query result so later we can filter (eg discard congressperson replies), So I'd go with something like that:

document_id | tweet_id | reply_id | reply_user | suspicion_confirmed

Where:

• document_id: reimbursement document_id
• tweet_id: original Rosie's tweet ID
• reply_id: reply tweet ID
• reply_user: user who replied to Rosie
• classification: true or false

[cuducos]

Complementing my previous message, a roadmap for a possible implementation would be:

1. check all new replies @RosieDaSerenata has got
2. check if in the in_reply_to_… field we have a @RosieDaSerenata tweet with a valid Jarbas URL
3. fetch the original tweet and extract the document_id from id
4. scan the reply for a certain hashtag (#falsoPositivo or #RosieAcertaOutraVez for example)
5. persist that data in a dataset or database

[vmesel]

@cuducos I was saying that there is no reply endpoint, I do know that, there is a mentions endpoint and the example (that you've said it's old, but it's still gold) given uses the same logic you said.

Stack Overflow example:

From status/show you can find the user's id. Then statuses/mentions_timeline will return a list of status for a user. Just parse that return looking for a in_reply_to_status_id matching the original tweet's id.

Why should we have individual records for replies if we just want to classify the receipt? This consumes lots of unnecessary space on the database.

We could do an update on the line referencing this tweet/document id.

Another thing on this request is: how can we recover the Jarbas document ID without making a web request, this would be time and money consuming (using the network costs of DO, or AWS).

[cuducos]

Why should we have individual records for replies if we just want to classify the receipt?

As I've said:

so later we can filter (eg discard congressperson replies)

[cuducos]

Oops; sent before finishing… so here we go again.

Why should we have individual records for replies if we just want to classify the receipt?

As I've said:

so later we can filter (eg discard congressperson replies)

I cannot predict which filters are useful (bots could be created in massive attacks to poor Rosie, for example), so I think storing them all is the best choice.

how can we recover the Jarbas document ID without making a web request

Does the text in the tweet API is the full URL or Twitter shortened version? If it's shortened, there's no wayout. If it's not we can parse the URL (it's always https://jarbas.serenatadeamor.org/#/documentId/<document_id>).

[Irio]

@cuducos The URLs are shortened. You get the full ones making a request to https://t.co/.

[vmesel]

@Irio so we have a great bandwith problem here! Depending on the quantity and how we are going to organize the requests to http://t.co/ it might think its a DDOS attack and block our IPs.

[Irio]

@vmesel This is not a problem since the document_id's of all posts are stored in a database. Check again the link I posted.

[cuducos]

we have a great bandwith problem here!

Do we? Share the code and we can help you.

[cuducos]

Depending on the quantity and how we are going to organize the requests to http://t.co/ it might think its a DDOS attack and block our IPs.

This is not a bandwidth problem: this is a software problem. If this is a problem we just pause between requests, for example, as we do here.