Techie BabajiLearn to Code. Build Real Skills.
Node.js · Events & Data

Streams

Reading a 2 GB log file into memory would crash your server. Streams process data in small chunks as it flows — files, HTTP bodies, compression — using little memory no matter how big the data is.

Intermediate 6 min Lesson 13 of 37Examples target Node.js 22 LTS.
Before you start:BuffersEvents
0/37 in Node.js
On this page · 6 min · Intermediate

On this page

  1. Learning objectives
  2. Explanation
  3. Syntax
  4. Code example
  5. Step by step
  6. Real-world use
  7. Common mistakes
  8. Best practices
  9. Remember this
  10. Practice exercise
  11. Mini quiz
  12. Related topics
  13. My notes
Reading time
6 min
Difficulty
Intermediate
Updated
Oct 28, 2026

Related tutorials

Interview questions

Learning objectives#

  • Explain readable, writable, duplex and transform streams
  • Connect streams safely with pipeline()
  • Write a custom transform stream
  • Stream files to HTTP responses

Explanation#

Readable streams produce data (fs.createReadStream, an HTTP request). Writable streams consume it (fs.createWriteStream, an HTTP response). Transform streams change data passing through (gzip, CSV parsing).

Data moves in chunks (Buffers or strings). Only a small window is in memory at once.

pipeline(source, ...transforms, destination) from node:stream/promises connects streams, handles backpressure (pausing a fast reader when the writer is slow) and propagates errors and cleanup. Prefer it over .pipe().

Readable streams are async iterable: for await (const chunk of stream) { ... } is an easy way to consume them.

Servers stream naturally: pipe a file read stream into res to send downloads without loading the file into memory.

Syntax#

JavaScript Illustrative
import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';


await pipeline(createReadStream('big.log'), createGzip(), createWriteStream('big.log.gz'));

Code example: Filter a large log file line by line and gzip it#

JavaScriptstreams.js Illustrative
import { createReadStream, createWriteStream } from 'node:fs';
import { mkdtemp, writeFile, stat, rm } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';
import { createInterface } from 'node:readline';
import { createGzip } from 'node:zlib';
import path from 'node:path';
import os from 'node:os';


const dir = await mkdtemp(path.join(os.tmpdir(), 'streams-'));
const logFile = path.join(dir, 'app.log');


// Create a "big" log file: 100,000 lines, 1 in 10 is an ERROR
const lines = Array.from({ length: 100_000 }, (_, i) => (i % 10 === 0 ? `ERROR order ${i} failed` : `INFO request ${i} ok`));
await writeFile(logFile, lines.join('\n') + '\n');


// A transform stream that keeps only ERROR lines (handles lines split across chunks)
function onlyErrors() {
  let leftover = '';
  return new Transform({
    transform(chunk, _encoding, done) {
      const parts = (leftover + chunk.toString()).split('\n');
      leftover = parts.pop();
      done(null, parts.filter(l => l.startsWith('ERROR')).map(l => l + '\n').join(''));
    },
    flush(done) {
      done(null, leftover.startsWith('ERROR') ? leftover + '\n' : '');
    },
  });
}


const errorsFile = path.join(dir, 'errors.log.gz');
await pipeline(createReadStream(logFile), onlyErrors(), createGzip(), createWriteStream(errorsFile));


const original = (await stat(logFile)).size;
const compressed = (await stat(errorsFile)).size;
console.log('Original log:', (original / 1024).toFixed(0), 'KB');
console.log('Errors only, gzipped:', compressed < original / 20 ? 'less than 5% of the original' : 'larger than expected');


// Count lines with readline, which reassembles lines split across chunks
let errorCount = 0;
for await (const line of createInterface({ input: createReadStream(logFile) })) {
  if (line.startsWith('ERROR')) errorCount++;
}
console.log('ERROR lines:', errorCount);


await rm(dir, { recursive: true, force: true });
Expected output
Original log: 2167 KB
Errors only, gzipped: less than 5% of the original
ERROR lines: 10000

This example is illustrative. Copy or download it and run it in your own environment; the expected output is shown above.

Step-by-step explanation#

  1. createReadStream reads the 2 MB file in 64 KB chunks instead of all at once.
  2. Counting matches chunk by chunk would miss lines split across chunks (it really finds 9,999 here). readline reassembles whole lines.
  3. onlyErrors is a Transform: it keeps the incomplete last line of each chunk for the next chunk, and flush handles the very end.
  4. createGzip compresses the filtered lines; createWriteStream writes them to disk.
  5. pipeline wires everything together, handles backpressure and rejects if any step fails.
  6. for await over a readline interface consumes the file line by line.

Real-world use case#

A reporting API exports 500,000 orders as CSV: it streams rows from MySQL through a CSV transform and gzip straight into the HTTP response, using about 30 MB of memory instead of several GB.

Common mistakes#

readFile on huge files

Loading the whole file into memory can crash the process. Stream it.

.pipe() without error handling

Errors do not propagate through pipe chains and streams may leak. Use pipeline().

Assuming chunks are whole lines

Chunk boundaries are arbitrary. Buffer partial lines (or use readline).

Best practices#

  • Stream anything large
  • Use pipeline() from node:stream/promises
  • Handle partial records at chunk boundaries
  • Stream file downloads to HTTP responses

Remember this#

Readable → Transform → Writable.

pipeline() = pipe + errors + backpressure + cleanup.

Big data? Stream it, do not load it.

Practice exercise#

Your task: Write a script that copies a file and converts it to upper case on the way, using pipeline and a Transform.

Need a hint?
new Transform({ transform(chunk, enc, done) { done(null, chunk.toString().toUpperCase()); } }).
Show solution
JavaScript Illustrative
import { createReadStream, createWriteStream } from 'node:fs';
import { writeFile, readFile, mkdtemp, rm } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';
import path from 'node:path';
import os from 'node:os';


const dir = await mkdtemp(path.join(os.tmpdir(), 'upper-'));
await writeFile(path.join(dir, 'in.txt'), 'hello streams');


const upper = new Transform({
  transform(chunk, _enc, done) { done(null, chunk.toString().toUpperCase()); },
});
await pipeline(createReadStream(path.join(dir, 'in.txt')), upper, createWriteStream(path.join(dir, 'out.txt')));


console.log(await readFile(path.join(dir, 'out.txt'), 'utf8'));   // HELLO STREAMS
await rm(dir, { recursive: true });

Mini quiz#

Streams quizMultiple choice Question 1 of 3

Which stream type modifies data as it passes through?

Score so far: 0/0

Want more? Practise all Node.js MCQs · Node.js interview questions

My notes#

Saved in this browser.

Progress is saved in this browser. Sign in to sync.