1 |
3a515b92
|
cagy
|
# Async-Limiter
|
2 |
|
|
|
3 |
|
|
A module for limiting concurrent asynchronous actions in flight. Forked from [queue](https://github.com/jessetane/queue).
|
4 |
|
|
|
5 |
|
|
[![npm](http://img.shields.io/npm/v/async-limiter.svg?style=flat-square)](http://www.npmjs.org/async-limiter)
|
6 |
|
|
[![tests](https://img.shields.io/travis/STRML/async-limiter.svg?style=flat-square&branch=master)](https://travis-ci.org/STRML/async-limiter)
|
7 |
|
|
[![coverage](https://img.shields.io/coveralls/STRML/async-limiter.svg?style=flat-square&branch=master)](https://coveralls.io/r/STRML/async-limiter)
|
8 |
|
|
|
9 |
|
|
This module exports a class `Limiter` that implements some of the `Array` API.
|
10 |
|
|
Pass async functions (ones that accept a callback or return a promise) to an instance's additive array methods.
|
11 |
|
|
|
12 |
|
|
## Motivation
|
13 |
|
|
|
14 |
|
|
Certain functions, like `zlib`, have [undesirable behavior](https://github.com/nodejs/node/issues/8871#issuecomment-250915913) when
|
15 |
|
|
run at infinite concurrency.
|
16 |
|
|
|
17 |
|
|
In this case, it is actually faster, and takes far less memory, to limit concurrency.
|
18 |
|
|
|
19 |
|
|
This module should do the absolute minimum work necessary to queue up functions. PRs are welcome that would
|
20 |
|
|
make this module faster or lighter, but new functionality is not desired.
|
21 |
|
|
|
22 |
|
|
Style should confirm to nodejs/node style.
|
23 |
|
|
|
24 |
|
|
## Example
|
25 |
|
|
|
26 |
|
|
``` javascript
|
27 |
|
|
var Limiter = require('async-limiter')
|
28 |
|
|
|
29 |
|
|
var t = new Limiter({concurrency: 2});
|
30 |
|
|
var results = []
|
31 |
|
|
|
32 |
|
|
// add jobs using the familiar Array API
|
33 |
|
|
t.push(function (cb) {
|
34 |
|
|
results.push('two')
|
35 |
|
|
cb()
|
36 |
|
|
})
|
37 |
|
|
|
38 |
|
|
t.push(
|
39 |
|
|
function (cb) {
|
40 |
|
|
results.push('four')
|
41 |
|
|
cb()
|
42 |
|
|
},
|
43 |
|
|
function (cb) {
|
44 |
|
|
results.push('five')
|
45 |
|
|
cb()
|
46 |
|
|
}
|
47 |
|
|
)
|
48 |
|
|
|
49 |
|
|
t.unshift(function (cb) {
|
50 |
|
|
results.push('one')
|
51 |
|
|
cb()
|
52 |
|
|
})
|
53 |
|
|
|
54 |
|
|
t.splice(2, 0, function (cb) {
|
55 |
|
|
results.push('three')
|
56 |
|
|
cb()
|
57 |
|
|
})
|
58 |
|
|
|
59 |
|
|
// Jobs run automatically. If you want a callback when all are done,
|
60 |
|
|
// call 'onDone()'.
|
61 |
|
|
t.onDone(function () {
|
62 |
|
|
console.log('all done:', results)
|
63 |
|
|
})
|
64 |
|
|
```
|
65 |
|
|
|
66 |
|
|
## Zlib Example
|
67 |
|
|
|
68 |
|
|
```js
|
69 |
|
|
const zlib = require('zlib');
|
70 |
|
|
const Limiter = require('async-limiter');
|
71 |
|
|
|
72 |
|
|
const message = {some: "data"};
|
73 |
|
|
const payload = new Buffer(JSON.stringify(message));
|
74 |
|
|
|
75 |
|
|
// Try with different concurrency values to see how this actually
|
76 |
|
|
// slows significantly with higher concurrency!
|
77 |
|
|
//
|
78 |
|
|
// 5: 1398.607ms
|
79 |
|
|
// 10: 1375.668ms
|
80 |
|
|
// Infinity: 4423.300ms
|
81 |
|
|
//
|
82 |
|
|
const t = new Limiter({concurrency: 5});
|
83 |
|
|
function deflate(payload, cb) {
|
84 |
|
|
t.push(function(done) {
|
85 |
|
|
zlib.deflate(payload, function(err, buffer) {
|
86 |
|
|
done();
|
87 |
|
|
cb(err, buffer);
|
88 |
|
|
});
|
89 |
|
|
});
|
90 |
|
|
}
|
91 |
|
|
|
92 |
|
|
console.time('deflate');
|
93 |
|
|
for(let i = 0; i < 30000; ++i) {
|
94 |
|
|
deflate(payload, function (err, buffer) {});
|
95 |
|
|
}
|
96 |
|
|
t.onDone(function() {
|
97 |
|
|
console.timeEnd('deflate');
|
98 |
|
|
});
|
99 |
|
|
```
|
100 |
|
|
|
101 |
|
|
## Install
|
102 |
|
|
|
103 |
|
|
`npm install async-limiter`
|
104 |
|
|
|
105 |
|
|
## Test
|
106 |
|
|
|
107 |
|
|
`npm test`
|
108 |
|
|
|
109 |
|
|
## API
|
110 |
|
|
|
111 |
|
|
### `var t = new Limiter([opts])`
|
112 |
|
|
Constructor. `opts` may contain inital values for:
|
113 |
|
|
* `t.concurrency`
|
114 |
|
|
|
115 |
|
|
## Instance methods
|
116 |
|
|
|
117 |
|
|
### `t.onDone(fn)`
|
118 |
|
|
`fn` will be called once and only once, when the queue is empty.
|
119 |
|
|
|
120 |
|
|
## Instance methods mixed in from `Array`
|
121 |
|
|
Mozilla has docs on how these methods work [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array).
|
122 |
|
|
### `t.push(element1, ..., elementN)`
|
123 |
|
|
### `t.unshift(element1, ..., elementN)`
|
124 |
|
|
### `t.splice(index , howMany[, element1[, ...[, elementN]]])`
|
125 |
|
|
|
126 |
|
|
## Properties
|
127 |
|
|
### `t.concurrency`
|
128 |
|
|
Max number of jobs the queue should process concurrently, defaults to `Infinity`.
|
129 |
|
|
|
130 |
|
|
### `t.length`
|
131 |
|
|
Jobs pending + jobs to process (readonly).
|