Remove redundant whitespace from pdf text

[seocahill]

It seems to be significant AFAICT?

[13] pry(main)> chunk = pages[3].text
=> "                                       INTRODUCTION\n\n\n\n   THEIRISH LANGUAGE\n\n   Irish is one of the many languages spoken aross Europe and as far east as India, that trace\ntheir descent from Indo-European, a hypothetical ancestor-language thought to have been\nspoken more than 4,500 years ago. Irish belongs to the Celtic branch of the Indo-European\nfamily, as the diagram below shows. It and three other members of this branch - Scottish\nGaelic,Welsh and Breton - are today alive ascommunity languages.\n\n   The form of Celtic that was to become Irish was brought to Ireland by the invading Gaels\n- about300 B.C. according to some scholars. Later it spread to Scotland and the Isle of Man.\nScottish Gaelic and Manx gradually separated from Irish (and,more slowly,from each other),\nand they can be thought of as distinct languages from the seventeenth century onwards. The\nterm 'Gaelic* may beused todenote all three.\n\n\n   It appears that the early Irish learned the art of writing at about the timeof their conversion\nto Christianity, in the fifth century. After that, the language can be seen to go throughfour\nstages of continuous historical development, asfar as its written form is concerned: Old Irish\n(approximately A.D. 600- 900), Middle Irish (c. 900- 1200), Early Modern Irish (c. 1200 -\n1650), and Modern Irish. Throughout this development Irish borrowed words from other\nlanguages it came into contact with (pre-eminently from Latin, from Norse, from Anglo-\nNorman (a dialect of French), andfrom English.\n\n   From the earliest times Irish has been cultivated for literature and learning. It in fact\npossesses one of the oldest literatures in Europe.\n\n\n\n                                       INDOEUROPEAN\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n                                                                                          Sanskrit\n                                                                                          Hindi\n                                                                                          Urdu\n                                                                                          Bengali\n                                                                                          Punjabi\n               GERMANIC\n                                                                                          Singhalese\n                                         CELTIC             Latin, etc.                     etc.\n       Icelandic     English\n       Faroese       Frisian                                 Portuguese\n       Norwegian     Dutch                       \\\n       Swedish       German        Irish                     Spanish\n                                   Scottish-     Welsh       Catalan\n       Danish          etc.          Gaelic      Breton      French\n         etc.                      Manx          Cornish     Italian\n                                                             Rumanian\n                                                               etc."
[14] pry(main)> enc.encode(chunk).length
=> 498
[15] pry(main)> enc.encode(chunk.gsub(/\s+/, ' ')).length
=> 415

[seocahill]

...but I'm very new to this. Thanks for the gem btw - great work 👍

[andreibondarev]

@seocahill I'm wondering if removing extra white space and new lines should be the responsibility of a chunker and not processors?