Pointers on Reading+Listening

Pointers to research papers and books, online articles, and projects exploring reading+listening (aka reading-while-listening aka bimodal reading aka multimodal reading aka read-aloud reading) for literacy building.

Research Papers

(This is a very rough selection among hundreds, if not thousands, of academic papers about reading+listening, and it is geared towards the theme of literacy building.)

1. Benavides-Varela et al., Newborn's brain activity signals the origin of word memories, 2012
2. Bird et al., The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling, 2002
3. Bisson et al., Processing of native and foreignlanguage subtitles in films: An eyetracking study, 2014
4. Boyle et al., Effects of Audio Text on the Acquisition of Secondary-Level Content by Students with Mild Disabilities, 2003
5. Chang, The Effect of Reading While Listening to Audiobooks: Listening Fluency and Vocabulary Gain, 2011
6. Dijkstra et al., Grapheme context effects on phonemic processing, 1989
7. Dilani et al., The Effects of Audiobooks on the Psychosocial Adjustment of Pre-adolescents and Adolescents with Dyslexia, 2008
8. Frost et al., Can speech perception be influenced by simultaneous presentation of print?, 1988
9. Holobow et al., Pairing script and dialogue: combinations that show promise for second or foreign language learning, 1984
10. Kothari et al., Same Language Subtitling of Bollywood Film Songs on TV: Effects on Literacy, 2003
11. Kothari et al., Same language subtitling: a butterfly for literacy?, 2002
12. Lionetti et al., A comparison of the effects of two rates of listening while reading on oral reading fluency and reading comprehension, 2004
13. Lund, A comparison of second language listening and reading comprehension, 1991
14. McCall et al., Same-Language-Subtitling (SLS): Using Subtitled Music Video for Reading Growth, 2009
15. McMahon, Development of reading-while-listening skills in the primary grades, 1983
16. Montali et al., Bimodal reading: benefits of a talking computer for average and less skilled readers, 1996
17. Pasnik, iPod in Education: The Potential for Teaching and Learning, 2007
18. Perego et al., The Cognitive Effectiveness of Subtitle Processing, 2010
19. Rose et al., Plato Revisited: Learning Through Listening in the Digital World, 2007
20. Schmithorst et al., Cognitive modules utilized for narrative comprehension in children: a functional magnetic resonance imaging study, 2006
21. Schmithorst et al., Development of effective connectivity for narrative comprehension in children, 2007
22. Skinner et al., Effects of Different Listening-While-Reading Rates on Comprehension in Secondary Students with Reading Deficits, 1998
23. Taguchi et al., Developing reading fluency in EFL: How assisted repeated reading and extensive reading affect fluency development, 2004
24. Teja, The effects of the read 180 program on oral reading fluency, linguistic comprehension, and reading comprehension with secondary special education students, 2014
25. Van Bon et al., A comparison of three methods of reading-while-listening, 1991
26. Winke et al., The effects of captioning videos used for foreign language listening activities, 2010
27. Winn et al., The Effects of Listening while Reading and Repeated Reading on the Reading Fluency of Adult Learners, 2006
28. Woodall, Simultaneous Listening and Reading in ESL: Helping Second Language Learners Read (and Enjoy Reading) More Efficiently, 2008
29. Zientarski et al., Reading Aloud to Low Achieving Secondary Students, 1994

Books

1. Falghera, Lettura+Ascolto. Come migliorare l'apprendimento linguistico, emotivo ed empatico con gli audiolibri (in Italian), 2013
2. Sousa, How the Brain Learns to Read, 2014
3. Wolf, Dyslexia, Fluency, and the Brain, 2001
4. Wolf, Proust and the Squid: The Story and Science of the Reading Brain, 2007

Online Articles

1. Brady-Myerov, An overview on using closed captioning for improving literacy
2. Garrish, Some Nuances of Media Overlays
3. MLOL, Gli audio-ebook: libri da leggere e da ascoltare (in Italian)
4. Pettarin, Audio-eBooks: using Media Overlays in reflowable EPUB 3 eBooks
5. Pettarin, How To Create EPUB 3 Read Aloud eBooks
6. Pettarin, On Audio eBooks and Multimodal Reading
7. ReadBeyond, Improving Literacy with Audio-eBooks
8. Yellin, VOICEtext – Text Synched with Audiobooks Improves Reading

Projects (Software/Companies)

BookTrack

Booktrack offers a new content creation and distribution platform that turns reading into an immersive movie-like experience. Booktrack’s patented technology lets anyone add a synchronized movie-style soundtrack to an e-book or other digital text content, with the audio paced to each individual’s reading speed.

The entire platform is proprietary. Independent research shows (1, 2) a modest increase in retention, and a more significant increase in engagement, compared to standard reading. They recently got a series B VC round of 5M USD, following a 3M USD first VC round.

DAISY Consortium

The DAISY Consortium is a global partnership of organizations committed to creating the best way to read and publish, for everyone. We strive to make e-books accessible for everyone by contributing to the development of open accessible standards (DAISY, EPUB 3) and supporting inclusive publishing efforts around the world.

A consortium developing standards for talking books, and member of the IDPF, developing the EPUB 3 Media Overlays specification for Audio-eBooks with synchronized audio/text. They also have several free tools for creating audiobooks and Audio-eBooks.

Kindle Immersion Reading

Immerse you in a story by narrating and highlighting the text as you read. It sparks an extra connection that boosts engagement, comprehension, and retention, taking you deeper into the book.

After the acquisition of Audible by Amazon, they integrated the Audible audiobooks into the Kindle platform. Now you can read+listen books for which they computed the synchronization timings and/or switch consumption modality.

LearningAlly

Audiobooks are proven to help students with reading disabilities become better readers and learners. They can be a powerful tool to support comprehension, boost confidence, save time on schoolwork and ultimately lead to better outcomes.Audiobooks are proven to help students with reading disabilities become better readers and learners. They can be a powerful tool to support comprehension, boost confidence, save time on schoolwork and ultimately lead to better outcomes.

Several commercial products, including Audio-eBooks with synchronized highlighting.

ListenCurrent

Listen Current makes it easy to bring authentic voices and compelling non-fiction stories to the classroom. We curate the best of public radio to keep teaching connected to the real world and build student listening skills at the same time.

Create curriculum-integrated assignments and other classroom materials, using public radio broadcasts, with synchronized transcripts, as the main ingredient to boost text comprehension and other word literacy abilities.

MeeGenius

Captivate your child’s imagination by giving them access to hundreds of read-along books with MeeGenius, turning your mobile device into an expansive library of digital books for children up to 8 years old. All MeeGenius books are available with read-along word highlighting, rich illustrations, and engaging story narration.

Another well funded (2.4M USD) entirely proprietary platform for read-along Audio-eBooks, targeted at kids.

PlanetRead

PlanetRead is a not-for-profit organization registered in CA, USA and in India. It is dedicated to reading and literacy development around the world and has a solid track record of work in India. PlanetRead was originally created around the idea of Same Language Subtitling (SLS), now a globally recognized innovation for mass literacy and reading development on TV.

A well-known project using closed captioning for literacy programs over popular media like TV in developing countries.

ReadBeyond

We love great stories We think everybody should be able to share one with fellow human beings. We are a research lab developing software for creating, distributing and enjoying digital contents.

(Full disclosure: I was one of the founders and CEO of RB.)

The three main "products" of ReadBeyond are:

1. the aeneas Python package, which automatically synchronizes audio and text;
2. the Menestrello Android/iOS app, which reads EPUB 3 files and it is still the best app for EPUB 3 Audio-eBooks
3. the collection of 111 English Short Stories, freely downloadable EPUB 3 Audio-eBooks

Unfortunately, after an initial "seed" investment (burnt in tax consultant fees and crazy Italian taxes/bureaucracy), the company had to be put down in June 2015. The project survives as a "spare time" project of yours truly.

ReadSpeaker

Use the power of ReadSpeaker text to speech to give a voice to your websites, mobile apps, digital books, elearning materials, documents, and more!

ReadSpeaker offers (synthetic) text-to-speech services for augmenting Web pages and educational material. They have online a white paper entitled The benefits of bimodal content presentation.

SIL Scripture App Builder

Scripture App Builder helps you to build customized Scripture apps for Android smartphones and tablets. You specify the Scripture files to use, the app name, the fonts, the colors, the about box information, the audio and the icons. Scripture App Builder will package everything together and build the customized app for you. You can then install it on your Android device, send it to others by Bluetooth, share it on microSD memory cards and publish it to app stores on the Internet.

This tool allows you to build Android apps or export EPUB 3 files. Currently the timing files must be authored manually prior building the app or output files, but engineers at SIL have successfully tested our package aeneas successfully, and this tool has export-to-aeneas + input-from-aeneas functions to integrate with it. (Nice job, SIL!)