Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions

Aldo Hernandez-Suarez; Gabriel Sanchez-Perez; Karina Toscano-Medina; Rocio Toscano-Medina; Victor Martinez-Hernandez; Jesus Olivares-Mercado; Hector Perez-Meana; Victor Sanchez

doi:10.3233/978-1-61499-900-3-453

Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions

Aldo Hernandez-Suarez, Gabriel Sanchez-Perez, Karina Toscano-Medina, Rocio Toscano-Medina, Victor Martinez-Hernandez, Jesus Olivares-Mercado, Hector Perez-Meana, Victor Sanchez

Escuela Superior de Ingeniería Mecánica y Eléctrica (ESIME), Unidad Culhuacán

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Retrieving information from social networks is a first and primordial step in many data analysis fields such as Natural Language Processing and Machine Learning. Important data science tasks rely on historical data gathering for further predictive results. Recent works use public platforms for collecting public streams of information like Twitter API, which allows querying chronological tweets from periods no longer than three weeks. In this paper, we present Twitter Scrapy, a new methodology for collecting historical tweets from time periods of arbitrary duration using web scraping techniques that bypass Twitter API restrictions.

Original language	English
Title of host publication	New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018
Editors	Hamido Fujita, Enrique Herrera-Viedma
Publisher	IOS Press BV
Pages	453-462
Number of pages	10
ISBN (Electronic)	9781614998990
DOIs	https://doi.org/10.3233/978-1-61499-900-3-453
State	Published - 2018
Event	17th International Conference on New Trends in Intelligent Software Methodology Tools and Techniques, SoMeT 2018 - Granada, Spain Duration: 26 Sep 2018 → 28 Sep 2018

Publication series

Name	Frontiers in Artificial Intelligence and Applications
Volume	303
ISSN (Print)	0922-6389
ISSN (Electronic)	1879-8314

Conference

Conference	17th International Conference on New Trends in Intelligent Software Methodology Tools and Techniques, SoMeT 2018
Country/Territory	Spain
City	Granada
Period	26/09/18 → 28/09/18

Keywords

Twitter bots
Twitter scrapy
Web crawling
Web scraping
Web spiders

Access to Document

10.3233/978-1-61499-900-3-453

Cite this

Hernandez-Suarez, A., Sanchez-Perez, G., Toscano-Medina, K., Toscano-Medina, R., Martinez-Hernandez, V., Olivares-Mercado, J., Perez-Meana, H., & Sanchez, V. (2018). Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions. In H. Fujita, & E. Herrera-Viedma (Eds.), New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018 (pp. 453-462). (Frontiers in Artificial Intelligence and Applications; Vol. 303). IOS Press BV. https://doi.org/10.3233/978-1-61499-900-3-453

Hernandez-Suarez, Aldo ; Sanchez-Perez, Gabriel ; Toscano-Medina, Karina et al. / Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions. New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018. editor / Hamido Fujita ; Enrique Herrera-Viedma. IOS Press BV, 2018. pp. 453-462 (Frontiers in Artificial Intelligence and Applications).

@inproceedings{dc53b901d5ab4d96b3d4fd636fb4849a,

title = "Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions",

abstract = "Retrieving information from social networks is a first and primordial step in many data analysis fields such as Natural Language Processing and Machine Learning. Important data science tasks rely on historical data gathering for further predictive results. Recent works use public platforms for collecting public streams of information like Twitter API, which allows querying chronological tweets from periods no longer than three weeks. In this paper, we present Twitter Scrapy, a new methodology for collecting historical tweets from time periods of arbitrary duration using web scraping techniques that bypass Twitter API restrictions.",

keywords = "Twitter bots, Twitter scrapy, Web crawling, Web scraping, Web spiders",

author = "Aldo Hernandez-Suarez and Gabriel Sanchez-Perez and Karina Toscano-Medina and Rocio Toscano-Medina and Victor Martinez-Hernandez and Jesus Olivares-Mercado and Hector Perez-Meana and Victor Sanchez",

note = "Publisher Copyright: {\textcopyright} 2018 The authors and IOS Press. All rights reserved.; 17th International Conference on New Trends in Intelligent Software Methodology Tools and Techniques, SoMeT 2018 ; Conference date: 26-09-2018 Through 28-09-2018",

year = "2018",

doi = "10.3233/978-1-61499-900-3-453",

language = "Ingl{\'e}s",

series = "Frontiers in Artificial Intelligence and Applications",

publisher = "IOS Press BV",

pages = "453--462",

editor = "Hamido Fujita and Enrique Herrera-Viedma",

booktitle = "New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018",

}

Hernandez-Suarez, A , Sanchez-Perez, G , Toscano-Medina, K, Toscano-Medina, R, Martinez-Hernandez, V, Olivares-Mercado, J , Perez-Meana, H & Sanchez, V 2018, Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions. in H Fujita & E Herrera-Viedma (eds), New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018. Frontiers in Artificial Intelligence and Applications, vol. 303, IOS Press BV, pp. 453-462, 17th International Conference on New Trends in Intelligent Software Methodology Tools and Techniques, SoMeT 2018, Granada, Spain, 26/09/18. https://doi.org/10.3233/978-1-61499-900-3-453

Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions. / Hernandez-Suarez, Aldo ; Sanchez-Perez, Gabriel ; Toscano-Medina, Karina et al.
New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018. ed. / Hamido Fujita; Enrique Herrera-Viedma. IOS Press BV, 2018. p. 453-462 (Frontiers in Artificial Intelligence and Applications; Vol. 303).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions

AU - Hernandez-Suarez, Aldo

AU - Sanchez-Perez, Gabriel

AU - Toscano-Medina, Karina

AU - Toscano-Medina, Rocio

AU - Martinez-Hernandez, Victor

AU - Olivares-Mercado, Jesus

AU - Perez-Meana, Hector

AU - Sanchez, Victor

PY - 2018

Y1 - 2018

N2 - Retrieving information from social networks is a first and primordial step in many data analysis fields such as Natural Language Processing and Machine Learning. Important data science tasks rely on historical data gathering for further predictive results. Recent works use public platforms for collecting public streams of information like Twitter API, which allows querying chronological tweets from periods no longer than three weeks. In this paper, we present Twitter Scrapy, a new methodology for collecting historical tweets from time periods of arbitrary duration using web scraping techniques that bypass Twitter API restrictions.

AB - Retrieving information from social networks is a first and primordial step in many data analysis fields such as Natural Language Processing and Machine Learning. Important data science tasks rely on historical data gathering for further predictive results. Recent works use public platforms for collecting public streams of information like Twitter API, which allows querying chronological tweets from periods no longer than three weeks. In this paper, we present Twitter Scrapy, a new methodology for collecting historical tweets from time periods of arbitrary duration using web scraping techniques that bypass Twitter API restrictions.

KW - Twitter bots

KW - Twitter scrapy

KW - Web crawling

KW - Web scraping

KW - Web spiders

UR - http://www.scopus.com/inward/record.url?scp=85063402491&partnerID=8YFLogxK

U2 - 10.3233/978-1-61499-900-3-453

DO - 10.3233/978-1-61499-900-3-453

M3 - Contribución a la conferencia

T3 - Frontiers in Artificial Intelligence and Applications

SP - 453

EP - 462

BT - New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018

A2 - Fujita, Hamido

A2 - Herrera-Viedma, Enrique

PB - IOS Press BV

T2 - 17th International Conference on New Trends in Intelligent Software Methodology Tools and Techniques, SoMeT 2018

Y2 - 26 September 2018 through 28 September 2018

ER -

Hernandez-Suarez A , Sanchez-Perez G , Toscano-Medina K, Toscano-Medina R, Martinez-Hernandez V, Olivares-Mercado J et al. Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions. In Fujita H, Herrera-Viedma E, editors, New Trends in Intelligent Software Methodologies, Tools and Techniques - Proceedings of the 17th International Conference, SoMeT 2018. IOS Press BV. 2018. p. 453-462. (Frontiers in Artificial Intelligence and Applications). doi: 10.3233/978-1-61499-900-3-453

Can twitter API be bypassed? A new methodology for collecting chronological information without restrictions

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this