Here we can give a brief introduction to our methodologies and the data we used to create the project.

To simplify the classification of quotes by whether they talk about disasters or not, we looked into transferring a model trained to classify whether a Tweet is talking about a disaster and using it to classify Quotebank quotes. To do so, we have started with this Kaggle challenge. Twitter has become an important communication channel in times of emergency, which is why the Kaggle challenge authors’ argument for more agencies being interested in programmatically monitoring Twitter for disaster detection (i.e. disaster relief organizations and news agencies). In that competition, data scientists are challenged to build a machine learning model that predicts which Tweets are about real disasters and which one’s aren’t. The challenge participants are provided with a dataset of 10,000 tweets that were hand classified, with a disclaimer of the dataset containing text that may be considered profane, vulgar, or offensive. To give a taste of how these tweets look like, consider the following examples:

These small texts are labeled as talking about a disaster in a general sense of disasters (accident, attack, battle, catastrophe, etc.) that we will refer to as general-disaster. We are, however, interested in natural disasters like earthquakes, but a successfully performing general-disaster classification model could still be used to narrow down the search for natural disasters in a dataset as big as Quotebank.

Among the numerous models submitted for the challenge, we have picked two with high test scores, one of which is a BERT transformer and the other a DISTILBERT. We will use BERT and DISTILBERT as their code names. We have modified them, performed a small hyperparameter search to find the best parameters, and trained the final model on all available data (train+test). Their respective test performance on the tweets dataset is shown below:

Given the big number of quotes in Quotebank (234'994'989), the computational time needed to predict the class for all Quotebank quotes was about 80 GPU days. Because of this, we applied the final models on a randomly chosen sample consisting of 0.1% of the whole dataset, which is still an enormous sample of 234’991 quotes and gives tight bounds for the resulting estimates

BERT has predicted 9.74% quotes to be disasters, and DISTILBERT slightly less - 8.89%. This, however, seems to be rather unrealistic given the nature of Quotebank quotes - does really about 1 out of 10 quotes talk about a general disaster? To evaluate the precision of these results, we have hand-labeled a subset of 400 positive classification results as true positives or false positives. In the hand labeling, we distinguished general disasters the models were trained on from natural disasters we are interested in. The resulting precisions are, however, quite low and are shown in the table below:

DISTILBERT had a far better generalization performance, as evaluated by precision, for both general disasters and natural disasters. The transfer learning didn’t succeed overall and the models were not robust to the change of text distribution from tweets to Quotebank quotes. This could have been expected given the significant differences in the texts that tweets have compared to quotes: tweets are much shorter, use loads of hashtags, URLs, and emojis, etc.

But going beneath just the numbers, we experienced many strange things when labeling the quotes. Firstly, BERT worked almost the opposite way from what we wanted it to, as if it deliberately decided to choose those quotes which have no connection to any disasters. They were about Formula-1, politics, and celebrity news. However, DISTILBERT was more promising. Even though it had many false positives, they would often be quotes about sports, but in the context of sports matches being wars, and people fighting for triumph. For example: “Our defense did a great job. There were some times where they could have been better, but I saw the puck well.” Also, it can be interesting that a lot of quotes were about veterans and their rehabilitation. Fires, health problems, cybercrime, and financial crises were also a frequent part of the false positives. The overall sentiment was also negative in these predictions, which is also something that is expected when predicting the topic of catastrophic events. All in all, none of the models performed well, but DISTILBERT had shown some robustness qualitatively.

Below are snippets of positives for BERT and DISTILBERT.

For the details of the transfer learning effort, you can head over to the relevant notebooks: tweets-bert, tweets-transfer-bert, tweets-transfer-distilbert, quotebank-sample.

What makes an earthquake worth to talk about?

All of the news portals are trying to find those events which reach the most readers. Natural disasters are a topic that interests many readers. Naturally, people don’t want to lag behind the world’s most important events, but that people are much more likely to read negative news than positive news, also contributes to that news portals are happy to bring down articles about these events. These tendencies are also true for celebrities, politicians and other media influencers whose thoughts and ideas are quoted in the media.

As for other news as well, the number of quotes in total and in the first period after the event is a meaningful indicator of its popularity in media. For the earthquakes we used this measurement to analyse the appearance of the disaster in the news. We also included the length between the first and last appearance of the earthquake in the quotes in the analysis, which happened to be less meaningful in this respect.

Due to Plotly’s auto-slideshow feature, the plots are not automatically rescaled. You can use Autoscale and Reset Axis buttons on the right top corner of the plot to better visualize the quote distributions of a particular disaster in the plot below.

As we would expect, people tend to talk about more in the first few months after the earthquake about the disaster. As for any news an event is interesting if it happened in the recent past. For the earthquakes, one would expect that its appearance in the news is longer that casual news, because of the post-earthquakes and the long time of recovery of an area. In our analysis, we could also observe a peak of the daily quotes after one year of the date of the event, this can be the result of the anniversary commemorations.

However, our main focus was that what can influence the media coverage of the earthquakes. To analyse this question we gathered a lot of data about the disaster itself and its location and examined the correlation between these indicators and the media coverage indicators. What we would expect that those factors which measures the human exposure.

As it can be seen in the heatmap below, we observed a massive correlation between the number of quotes and

• the number of death associated with the event
• the number of injured
• the reconstruction cost

As for the number of quotes in the first year the list expanded by

• aid contribution
• mortality rate between under 5 years olds
• number of days required to start a business