When working with large files or large amounts of data in Node.js, loading everything into memory at once is not practical. If we try to read a 2GB file using fs.readFile, the entire file gets loaded into memory, which can crash the application or slow it down significantly. Streams solve this by processing data in small chunks.
A stream is a sequence of data that is made available over time. Instead of waiting for all the data to be ready, we process it piece by piece as it arrives. This is similar to watching a video online. We do not wait for the entire video to download before watching it. The video plays while the rest is still loading.
\nNode.js has four types of streams: Readable, Writable, Duplex, and Transform.
\nA readable stream is a source of data. We can read from it chunk by chunk. A common example is reading a file:
\nconst fs = require(\"fs\");\n\nconst readStream = fs.createReadStream(\"largefile.txt\", { encoding: \"utf8\" });\n\nreadStream.on(\"data\", (chunk) => {\n console.log(\"Received chunk:\", chunk.length, \"bytes\");\n});\n\nreadStream.on(\"end\", () => {\n console.log(\"Finished reading\");\n});The data event fires every time a new chunk is available. The default chunk size is 64KB, but we can change it with the highWaterMark option.
A writable stream is a destination where we can send data. Writing to a file is a common example:
\nconst writeStream = fs.createWriteStream(\"output.txt\");\n\nwriteStream.write(\"First line\\n\");\nwriteStream.write(\"Second line\\n\");\nwriteStream.end(\"Last line\\n\");The real power of streams comes from piping, which connects a readable stream to a writable stream. Data flows from the source to the destination automatically:
\nconst readStream = fs.createReadStream(\"input.txt\");\nconst writeStream = fs.createWriteStream(\"output.txt\");\n\nreadStream.pipe(writeStream);This copies the file from input to output without loading the entire file into memory.
\nTransform streams sit between a readable and writable stream, modifying the data as it passes through. A practical example is compressing a file:
\nconst zlib = require(\"zlib\");\nconst gzip = zlib.createGzip();\n\nfs.createReadStream(\"input.txt\")\n .pipe(gzip)\n .pipe(fs.createWriteStream(\"input.txt.gz\"));The data flows from the file, through the gzip transform, and into the compressed output file.
\nStreams are useful whenever we deal with large data. Reading or writing large files, processing CSV or JSON data row by row, handling HTTP request and response bodies for file uploads or downloads, and piping data between services. For small data that fits comfortably in memory, the simpler readFile and writeFile methods are fine. But when the data size is unpredictable or potentially large, streams are the safer choice.