2-2-Detecting-Text-Language-by-Counting-Stop-Words.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detecting Text Language by Counting Stop Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on [Detecting Text Language With Python and NLTK by Alejandro Nolla](http://blog.alejandronolla.com/2013/05/15/detecting-text-language-with-python-and-nltk/)\n",
    "\n",
    "*Stop words* are words which are filtered out before processing because they are mostly grammatical as opposed to semantic in nature e.g. search engines remove words like 'want'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Tokenizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Yo man, it's time for you to shut yo' mouth! I ain't even messin' dawg.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "try:\n",
    "    from nltk.tokenize import wordpunct_tokenize # RE-based tokenizer which splits text on whitespace and punctuation (except for underscore)\n",
    "except ImportError:\n",
    "    print('[!] You need to install nltk (http://nltk.org/index.html)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Yo',\n",
       " 'man',\n",
       " ',',\n",
       " 'it',\n",
       " \"'\",\n",
       " 's',\n",
       " 'time',\n",
       " 'for',\n",
       " 'you',\n",
       " 'to',\n",
       " 'shut',\n",
       " 'yo',\n",
       " \"'\",\n",
       " 'mouth',\n",
       " '!',\n",
       " 'I',\n",
       " 'ain',\n",
       " \"'\",\n",
       " 't',\n",
       " 'even',\n",
       " 'messin',\n",
       " \"'\",\n",
       " 'dawg',\n",
       " '.']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tokens = wordpunct_tokenize(text)\n",
    "test_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other tokenizers e.g. `RegexpTokenizer` where you can enter your own regexp, `WhitespaceTokenizer` (similar to Python's `string.split()`) and `BlanklineTokenizer`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploring NLTK's stop words corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK comes with a corpus of stop words in various languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Stopwords Corpus  This corpus contains lists of stop words for several languages.  These are high-frequency grammatical words which are usually ignored in text retrieval applications.  They were obtained from: http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/snowball/stopwords/  The stop words for the Romanian language were obtained from: http://arlc.ro/resources/  The English list has been augmented https://github.com/nltk/nltk_data/issues/22  The German list has been corrected https://github.com/nltk/nltk_data/pull/49  A Kazakh list has been added https://github.com/nltk/nltk_data/pull/52  A Nepali list has been added https://github.com/nltk/nltk_data/pull/83  An Azerbaijani list has been added https://github.com/nltk/nltk_data/pull/100  A Greek list has been added https://github.com/nltk/nltk_data/pull/103    '"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopwords.readme().replace('\\n', ' ') # Since this is raw text, we need to replace \\n's with spaces for it to be readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['arabic',\n",
       " 'azerbaijani',\n",
       " 'danish',\n",
       " 'dutch',\n",
       " 'english',\n",
       " 'finnish',\n",
       " 'french',\n",
       " 'german',\n",
       " 'greek',\n",
       " 'hungarian',\n",
       " 'italian',\n",
       " 'kazakh',\n",
       " 'nepali',\n",
       " 'norwegian',\n",
       " 'portuguese',\n",
       " 'romanian',\n",
       " 'russian',\n",
       " 'spanish',\n",
       " 'swedish',\n",
       " 'turkish']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stopwords.fileids() # Most corpora consist of a set of files, each containing a piece of text. A list of identifiers for these files is accessed via fileids()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpus readers provide a variety of methods to read data from the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"αλλα\\nαν\\nαντι\\nαπο\\nαυτα\\nαυτεσ\\nαυτη\\nαυτο\\nαυτοι\\nαυτοσ\\nαυτουσ\\nαυτων\\nαἱ\\nαἳ\\nαἵ\\nαὐτόσ\\nαὐτὸς\\nαὖ\\nγάρ\\nγα\\nγα^\\nγε\\nγια\\nγοῦν\\nγὰρ\\nδ'\\nδέ\\nδή\\nδαί\\nδαίσ\\nδαὶ\\nδαὶς\\nδε\\nδεν\\nδι'\\nδιά\\nδιὰ\\nδὲ\\nδὴ\\nδ’\\nεαν\\nειμαι\\nειμαστε\\nειναι\\nεισαι\\nειστε\\nεκεινα\\nεκεινεσ\\nεκεινη\\nεκεινο\\nεκεινοι\\nεκεινοσ\\nεκεινουσ\\nεκεινων\\nενω\\nεπ\\nεπι\\nεἰ\\nεἰμί\\nεἰμὶ\\nεἰς\\nεἰσ\\nεἴ\\nεἴμι\\nεἴτε\\nη\\nθα\\nισωσ\\nκ\\nκαί\\nκαίτοι\\nκαθ\\nκαι\\nκατ\\nκατά\\nκατα\\nκατὰ\\nκαὶ\\nκι\\nκἀν\\nκἂν\\nμέν\\nμή\\nμήτε\\nμα\\nμε\\nμεθ\\nμετ\\nμετά\\nμετα\\nμετὰ\\nμη\\nμην\\nμἐν\\nμὲν\\nμὴ\\nμὴν\\nνα\\nο\\nοι\\nομωσ\\nοπωσ\\nοσο\\nοτι\\nοἱ\\nοἳ\\nοἷς\\nοὐ\\nοὐδ\\nοὐδέ\\nοὐδείσ\\nοὐδεὶς\\nοὐδὲ\\nοὐδὲν\\nοὐκ\\nοὐχ\\nοὐχὶ\\nοὓς\\nοὔτε\\nοὕτω\\nοὕτως\\nοὕτωσ\\nοὖν\\nοὗ\\nοὗτος\\nοὗτοσ\\nπαρ\\nπαρά\\nπαρα\\nπαρὰ\\nπερί\\nπερὶ\\nποια\\nποιεσ\\nποιο\\nποιοι\\nποιοσ\\nποιουσ\\nποιων\\nποτε\\nπου\\nποῦ\\nπρο\\nπροσ\\nπρόσ\\nπρὸ\\nπρὸς\\nπως\\nπωσ\\nσε\\nστη\\nστην\\nστο\\nστον\\nσόσ\\nσύ\\nσύν\\nσὸς\\nσὺ\\nσὺν\\nτά\\nτήν\\nτί\\nτίς\\nτίσ\\nτα\\nταῖς\\nτε\\nτην\\nτησ\\nτι\\nτινα\\nτις\\nτισ\\nτο\\nτοί\\nτοι\\nτοιοῦτος\\nτοιοῦτοσ\\nτον\\nτοτε\\nτου\\nτούσ\\nτοὺς\\nτοῖς\\nτοῦ\\nτων\\nτό\\nτόν\\nτότε\\nτὰ\\nτὰς\\nτὴν\\nτὸ\\nτὸν\\nτῆς\\nτῆσ\\nτῇ\\nτῶν\\nτῷ\\nωσ\\nἀλλ'\\nἀλλά\\nἀλλὰ\\nἀλλ’\\nἀπ\\nἀπό\\nἀπὸ\\nἀφ\\nἂν\\nἃ\\nἄλλος\\nἄλλοσ\\nἄν\\nἄρα\\nἅμα\\nἐάν\\nἐγώ\\nἐγὼ\\nἐκ\\nἐμόσ\\nἐμὸς\\nἐν\\nἐξ\\nἐπί\\nἐπεὶ\\nἐπὶ\\nἐστι\\nἐφ\\nἐὰν\\nἑαυτοῦ\\nἔτι\\nἡ\\nἢ\\nἣ\\nἤ\\nἥ\\nἧς\\nἵνα\\nὁ\\nὃ\\nὃν\\nὃς\\nὅ\\nὅδε\\nὅθεν\\nὅπερ\\nὅς\\nὅσ\\nὅστις\\nὅστισ\\nὅτε\\nὅτι\\nὑμόσ\\nὑπ\\nὑπέρ\\nὑπό\\nὑπὲρ\\nὑπὸ\\nὡς\\nὡσ\\nὥς\\nὥστε\\nὦ\\nᾧ\\n\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stopwords.raw('greek')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"αλλα αν αντι απο αυτα αυτεσ αυτη αυτο αυτοι αυτοσ αυτουσ αυτων αἱ αἳ αἵ αὐτόσ αὐτὸς αὖ γάρ γα γα^ γε για γοῦν γὰρ δ' δέ δή δαί δαίσ δαὶ δαὶς δε δεν δι' διά διὰ δὲ δὴ δ’ εαν ειμαι ειμαστε ειναι εισαι ειστε εκεινα εκεινεσ εκεινη εκεινο εκεινοι εκεινοσ εκεινουσ εκεινων ενω επ επι εἰ εἰμί εἰμὶ εἰς εἰσ εἴ εἴμι εἴτε η θα ισωσ κ καί καίτοι καθ και κατ κατά κατα κατὰ καὶ κι κἀν κἂν μέν μή μήτε μα με μεθ μετ μετά μετα μετὰ μη μην μἐν μὲν μὴ μὴν να ο οι ομωσ οπωσ οσο οτι οἱ οἳ οἷς οὐ οὐδ οὐδέ οὐδείσ οὐδεὶς οὐδὲ οὐδὲν οὐκ οὐχ οὐχὶ οὓς οὔτε οὕτω οὕτως οὕτωσ οὖν οὗ οὗτος οὗτοσ παρ παρά παρα παρὰ περί περὶ ποια ποιεσ ποιο ποιοι ποιοσ ποιουσ ποιων ποτε που ποῦ προ προσ πρόσ πρὸ πρὸς πως πωσ σε στη στην στο στον σόσ σύ σύν σὸς σὺ σὺν τά τήν τί τίς τίσ τα ταῖς τε την τησ τι τινα τις τισ το τοί τοι τοιοῦτος τοιοῦτοσ τον τοτε του τούσ τοὺς τοῖς τοῦ των τό τόν τότε τὰ τὰς τὴν τὸ τὸν τῆς τῆσ τῇ τῶν τῷ ωσ ἀλλ' ἀλλά ἀλλὰ ἀλλ’ ἀπ ἀπό ἀπὸ ἀφ ἂν ἃ ἄλλος ἄλλοσ ἄν ἄρα ἅμα ἐάν ἐγώ ἐγὼ ἐκ ἐμόσ ἐμὸς ἐν ἐξ ἐπί ἐπεὶ ἐπὶ ἐστι ἐφ ἐὰν ἑαυτοῦ ἔτι ἡ ἢ ἣ ἤ ἥ ἧς ἵνα ὁ ὃ ὃν ὃς ὅ ὅδε ὅθεν ὅπερ ὅς ὅσ ὅστις ὅστισ ὅτε ὅτι ὑμόσ ὑπ ὑπέρ ὑπό ὑπὲρ ὑπὸ ὡς ὡσ ὥς ὥστε ὦ ᾧ \""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stopwords.raw('greek').replace('\\n', ' ') # Better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\"]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stopwords.words('english')[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `.sents()` which returns sentences. However, in our particular case, this will cause an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'WordListCorpusReader' object has no attribute 'sents'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-10-e1185800ddd8>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mstopwords\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msents\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'greek'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'WordListCorpusReader' object has no attribute 'sents'"
     ]
    }
   ],
   "source": [
    "stopwords.sents('greek')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The erro is because the `stopwords` corpus reader is of type `WordListCorpusReader` so there are no sentences.\n",
    "It's the same for `.paras()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "444"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(stopwords.words(['english', 'greek'])) # There is a total of 444 Greek and English stop words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. The classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We loop through the list of stop words in all languages and check how many stop words our test text contains in each language. The text is then classified to be in the language in which it has the most stop words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'arabic': 0,\n",
       " 'azerbaijani': 0,\n",
       " 'danish': 3,\n",
       " 'dutch': 0,\n",
       " 'english': 8,\n",
       " 'finnish': 0,\n",
       " 'french': 2,\n",
       " 'german': 1,\n",
       " 'greek': 0,\n",
       " 'hungarian': 1,\n",
       " 'italian': 1,\n",
       " 'kazakh': 0,\n",
       " 'nepali': 0,\n",
       " 'norwegian': 3,\n",
       " 'portuguese': 1,\n",
       " 'romanian': 2,\n",
       " 'russian': 0,\n",
       " 'spanish': 1,\n",
       " 'swedish': 2,\n",
       " 'turkish': 0}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "language_ratios = {}\n",
    "\n",
    "test_words = [word.lower() for word in test_tokens] # lowercase all tokens\n",
    "test_words_set = set(test_words)\n",
    "\n",
    "for language in stopwords.fileids():\n",
    "    stopwords_set = set(stopwords.words(language)) # For some languages eg. Russian, it would be a wise idea to tokenize the stop words by punctuation too.\n",
    "    common_elements = test_words_set.intersection(stopwords_set)\n",
    "    language_ratios[language] = len(common_elements) # language \"score\"\n",
    "    \n",
    "language_ratios"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'english'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_rated_language = max(language_ratios, key=language_ratios.get) # The key parameter to the max() function is a function that computes a key. In our case, we already have a key so we set key to languages_ratios.get which actually returns the key.\n",
    "most_rated_language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ain', 'for', 'i', 'it', 's', 't', 'to', 'you'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_words_set.intersection(set(stopwords.words(most_rated_language))) # We can see which English stop words were found."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}