Data Streams with AWS Kinesis

When talking about messaging and asynchronous data processing, it is very common you hear about the jargon “data stream”. Let us start with the basic question.

The data stream is nothing but data that is continuously generated by thousands of data sources, which typically send data records simultaneously for processing. The data stream includes a wide variety of data such as log files generated in mobile or web applications, e-commerce purchases, in-game player activity, information from social networks, financial trading, and so on.

This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows. This is used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Let us discuss the related AWS offering ‘Kinesis’.

Per AWS documentation, “Amazon Kinesis Data Streams is a scalable and durable real-time data streaming service that can continuously capture gigabytes of data per second from hundreds of thousands of sources.”
You can continuously add various types of data such as clickstreams, application logs, and social media to an Amazon Kinesis data stream from hundreds of thousands of sources. Within seconds, the data will be available for your Amazon Kinesis Applications to read and process from the stream.

There will be producers generating huge data, Kinesis acts as an intermediate layer that lets consumers consume this data. Kinesis also has a concept of shard. You can think of it as a “box for data”, which can work independently. This is mainly for scaling purposes. To process more data, you need to configure more shards.

Apart from the stream of standard data (click-stream logs, database events, etc), we can have a custom producer component. A sample python snippet:

Next, we will see how to consume messages from shard/s in Kinesis.

Related posts:

The use of smart contracts on a blockchain was anticipated to “grow significantly,” particularly in industries such as government, insurance, banking, and more. This expansion takes into account the…

As the world becomes increasingly digital, so make the threats to our online security. Every day, there’s a new story in the news about a major data breach or cyber-attack. While it’s important to…

The staff augmentation business model has been transforming across various industries and domains. The reason behind its increasing usage is its highly competent model that bridges the companies’…