Streams are an amazing abstraction for many things. They let us pipe things together, transform them, and control their flow, all without having to buffer the contents into memory. However, sometimes you just want the full contents all at once. Maybe because you can't do what you need to as a stream, maybe because it's just easier for what you're doing. Screwing up collecting the results of a stream is one of the JavaScript common pitfalls.
Say, for example, you had a sample.txt file and you needed to know whether the contents are a palindrome, and if so how many characters long is it. Not the best as a streaming work load. So lets take a look at the easy hand-rolled implementation.
Using this sample.txt file:
a man a plan a canal panama
Our program looks like:
var fs = require('fs') var buffer = '' fs.createReadStream('./sample.txt') .on('data', function (chunk) { buffer += chunk }).on('end', function () { var noSpace = buffer.replace(/\s/g, '') var len = noSpace.length var isPalendrome = true for (var i = 0; i < (len/2); i++) { if (noSpace[i] !== noSpace[len-1-i]) { isPalendrome = false break; } } console.log(isPalendrome, len) }).on('error', function () { console.error('Oh, the humanity!') })
This works for our example, but it doesn't handle characters who have been split into different halves of a buffer, and it allocates a new string the size of the entire contents of the file for every chunk that comes in. It is also a lot of boiler plate code for a situation that comes of fairly often.
So, lets use a library to take care of the boiler plate. Max Ogden's 'concat-stream'. It takes a call back, and turns it into a stream you can pipe to. It buffers the contents of the stream internally, and then calls the function when it has the full results. It handles all kinds of things that you can stream, but you still have to watch for errors on the streams you pipe into it!
var fs = require('fs') var concat = require('concat-stream') fs.createReadStream('./sample.txt') .on('error', function () { console.error('Oh, the humanity!') }).pipe(concat(function (res) { var noSpace = res.toString().replace(/\s/g, '') var len = noSpace.length var isPalendrome = true for (var i = 0; i < (len/2); i++) { if (noSpace[i] !== noSpace[len-1-i]) { isPalendrome = false break; } } console.log(isPalendrome, len) }))
So, use concat-stream, 1million + downloads a month can't be wrong. I wrote my own module 'stream-cb' to handle many of the same use cases back in the wild west before we knew how to discover modules, if I'd know about concat-stream I would have used it instead.
So, quit writing janky buffering code on your own, and concat-stream all of the streams that you need to dam up and turn into lakes.