I want to build a server that could stream data from multiple endpoints simultaneously for analytics purposes.
Scenario: users registered on our platform will provide credentials of their IoT device. E.g https://stream.example.com/user1 & https://stream.example.com/user2
Our responsibility is to monitor the device log and status in order to generate reports.
Questions: Since each stream will keep a HTTP connection open, how can I make a node.js or ruby app to open serval HTTP stream?
Advertisement
Answer
Looking at your case I’d root for node.js too – keeping an open connection works well in asynchronous frameworks and Ruby doesn’t seem very fit (but then I’m not much into Ruby).
Using Scramjet this would be pretty easy, but keep in mind that both parties have to keep an open connection anyway, so it’s not the node app itself. I understand that your case is that your node app will actively open connections to a number of devices with the given credentials and these devices will then respond with a never ending stream over http, a bit like Twitter’s streaming API’s… then the implementation would look a little like this:
new scramjet.MultiStream(devicesList.map(
(device) => request.post(device, credentials)
.pipe(new StringStream())
.split("r?n") // or however you'd read your messages
.parse(JSON.parse) // or however you'd parse your data
.assign({deviceId: device.id}) // you'll probably need this
))
.mux(/* optionally an ordering function */)
// above there's a single stream with all the logs from all your devices.
If this is the other way round and the devices are calling the server, then take a look at scramjet-http-uristream or scramjet-http-post – it’s a simple sever that just streams all uri’s or posts that are posted to a server.
I hope this helps. 🙂