aws lambda使用_如何使用AWS Lambda为发布/订阅消息选择最佳事件源
aws lambda使用
by Yan Cui
崔燕
如何使用AWS Lambda为发布/订阅消息选择最佳事件源 (How to choose the best event source for pub/sub messaging with AWS Lambda)
AWS offers a wealth of options for implementing messaging patterns such as Publish/Subscribe
(often shortened to pub/sub) with AWS Lambda. In this article, we’ll compare and contrast some of these options.
AWS提供了许多选项来实现消息传递模式,例如使用AWS Lambda实现Publish/Subscribe
(通常简称为Publish/Subscribe
)。 在本文中,我们将比较和对比其中一些选项。
发布/订阅模式 (The pub/sub pattern)
Pub/Sub is a messaging pattern where publishers and subscribers are decoupled through an intermediary message broker (ZeroMQ, RabbitMQ, SNS, and so on).
发布/订阅是一种消息传递模式,其中发布者和订阅者通过中间消息代理(ZeroMQ,RabbitMQ,SNS等)解耦。
In the AWS ecosystem, the obvious candidate for the broker role is Simple Notification Service (SNS).
在AWS生态系统中,代理角色的明显候选者是简单通知服务(SNS)。
SNS will make three attempts for your Lambda function to process a message before sending it to a Dead Letter Queue (DLQ) if a DLQ is specified for the function. However, according to an analysis by the folks at OpsGenie, the number of retries can be as many as six.
如果为该函数指定了DLQ,SNS将在您的Lambda函数将消息发送到死信队列(DLQ)之前进行三次尝试。 但是,根据OpsGenie员工的分析 ,重试次数可以多达6次。
Another thing to consider is the degree of parallelism this setup offers. For each message, SNS will create a new invocation of your function. So if you publish 100 messages to SNS, then you can have 100 concurrent executions of the subscribed Lambda function.
要考虑的另一件事是此设置提供的并行度。 对于每条消息,SNS都会创建一个新的函数调用。 因此,如果将100条消息发布到SNS,则可以同时执行100个并发执行的所预订Lambda函数。
This is great if you’re optimising for throughput.
如果您要优化吞吐量,那就太好了。
However, we’re often constrained by the max throughput our downstream dependencies can handle — databases, S3, internal/external services, and so on.
但是,我们经常受到下游依赖项可以处理的最大吞吐量的约束 -数据库,S3,内部/外部服务等。
If the burst in throughput is short, then there’s a good chance the retries would be sufficient (there’s a randomized, exponential back off between retries too) and you won’t miss any messages.
如果吞吐量突发很短,那么重试就很有可能就足够了(重试之间也有随机的,指数的退避),并且您不会错过任何消息。
If the burst in throughput is sustained over a long period of time, then you can exhaust the max number of retries. At this point you’ll have to rely on the DLQ and possibly human intervention in order to recover the messages that couldn’t be processed the first time around.
如果吞吐量的爆发持续了很长时间,那么您可以用尽最大的重试次数。 此时,您必须依靠DLQ以及可能的人工干预来恢复第一次无法处理的消息。
Similarly, if the downstream dependency experiences an outage, then all messages received and retried during the outage are bound to fail.
同样,如果下游依赖项发生中断,则在中断期间接收和重试的所有消息都必然会失败。
You can also run into the Lambda limit on the number of concurrent executions in a region. Since this is an account-wide limit, it will also impact your other systems within the account that rely on AWS Lambda: APIs, event processing, cron jobs, and so on.
您还可以在区域中并发执行次数遇到Lambda限制 。 由于这是一个帐户范围的限制,因此也会影响帐户中依赖AWS Lambda的其他系统:API,事件处理,cron作业等。
SNS is also prone to suffer from temporal issues, like bursts in traffic, downstream outage, and so on. Kinesis, on the other hand, deals with these issues much better as described below:
SNS还容易受到时间问题的困扰,例如流量突发,下游中断等。 另一方面,Kinesis可以更好地处理这些问题,如下所述:
- The degree of parallelism is constrained by the number of shards, which can be used to amortize bursts in the message rate并行度受分片数量的限制,可用于分摊消息速率中的突发
- Records are retried until success is achieved, unless the outage lasts longer than the retention policy you have on the stream (the default is 24 hours). You will eventually be able to process the records将重试记录,直到成功为止,除非中断的持续时间长于您在流中拥有的保留策略(默认为24小时)。 您最终将能够处理记录
But Kinesis Streams is not without its own problems. In fact, from my experience using Kinesis Streams with Lambda, I have found a number of caveats that need to be understood in order to make effective use of the service.
但是Kinesis Streams并非没有自己的问题。 实际上,根据我将Kinesis Streams与Lambda结合使用的经验,我发现了一些需要注意的注意事项,以便有效地使用该服务。
You can read about these caveats in another article I wrote here.
您可以在我在这里写的另一篇文章中了解这些警告。
Interestingly, Kinesis Streams is not the only streaming option available on AWS. There is also DynamoDB Streams.
有趣的是,Kinesis Streams并不是AWS上唯一可用的流媒体选项。 还有DynamoDB流。
By and large, DynamoDB Streams + Lambda works the same way as Kinesis Streams + Lambda. Operationally, it does have some interesting twists:
总的来说,DynamoDB Streams + Lambda的工作方式与Kinesis Streams + Lambda相同。 在操作上,确实有一些有趣的转折:
- DynamoDB Streams auto scales the number of shardsDynamoDB Streams自动缩放分片数量
- If you’re processing DynamoDB Streams with AWS Lambda, then you don’t pay for the reads from DynamoDB Streams (but you still pay for the read and write capacity units for the DynamoDB table itself)如果您要使用AWS Lambda处理DynamoDB Streams,则无需为DynamoDB Streams的读取付费(但是您仍需为DynamoDB表本身的读写容量单位付费)
- Kinesis Streams offers the option to extend data retention to 7 days, but DynamoDB Streams doesn’t offer such an optionKinesis Streams提供了将数据保留期延长至7天的选项,但DynamoDB Streams不提供这种选择
The fact that DynamoDB Streams auto scales the number of shards can be a double-edged sword. On one hand, it eliminates the need for you to manage and scale the stream (or come up with home-baked auto scaling solutions). But on the other hand, it can also diminish the ability to amortize spikes in the load you pass on to downstream systems.
DynamoDB流自动扩展分片数量的事实可能是一把双刃剑。 一方面,它消除了您管理和扩展流(或提供自制的自动扩展解决方案 )的需要。 但另一方面,它也可能削弱摊销传递给下游系统的负载中的峰值的能力。
As far as I know, there is no way to limit the number of shards a DynamoDB stream can scale up to, which is something you’d surely consider when implementing your own auto scaling solution.
据我所知,没有办法限制DynamoDB流可以扩展的分片数量,这是您在实现自己的自动扩展解决方案时一定要考虑的问题。
I think the most pertinent question is, “what is your source of truth?”
我认为最相关的问题是: “what is your source of truth?”
Does a row being written in DynamoDB make it canon to the state of your system? This is certainly the case in most N-tier systems that are built around a database, regardless of whether it’s an RDBMS or NoSQL database.
在DynamoDB中写的行是否使其符合系统状态? 在围绕数据库构建的大多数N层系统中,无论是RDBMS数据库还是NoSQL数据库,都肯定是这种情况。
In an event-sourced system where state is modeled as a sequence of events (as opposed to a snapshot), the source of truth might well be the Kinesis stream. For example, as soon as an event is written to the stream, it’s considered canon to the state of the system.
在将事件建模为一系列事件(而不是快照)的事件源系统中,真理的源头很可能就是Kinesis流。 例如,一旦将事件写入流,就将其视为系统状态的标准。
Then, there’re other considerations around cost, auto-scaling, and so on.
然后,还有其他一些有关成本,自动缩放等方面的考虑。
From a development point of view, DynamoDB streams also have some limitations and shortcomings:
从开发的角度来看,DynamoDB流也有一些限制和不足:
- Each stream is limited to events from one table每个流仅限于一个表中的事件
- The records describe DynamoDB events and not events from your domain, which I’ve always felt creates a sense of dissonance when working with these events记录描述的是DynamoDB事件,而不是您域中的事件,在处理这些事件时,我总是觉得这会产生不和谐感
Excluding the cost of Lambda invocations for processing the messages, here are some cost projections for using SNS vs Kinesis Streams vs DynamoDB Streams as the broker. I’m making the assumption that throughput is consistent, and that each message is 1KB in size.
不包括用于处理消息的Lambda调用的成本,以下是一些使用SNS vs Kinesis Streams vs DynamoDB Streams作为代理的成本预测。 我假设吞吐量是一致的,每个消息的大小为1KB。
Monthly cost at 1 msg/s
每月费用为1 msg / s
Monthly cost at 1,000 msg/s
每月费用为1,000 msg / s
These projections should not be taken at face value. For starters, the assumption about a perfectly consistent throughput and message size is unrealistic, and you’ll need some headroom with Kinesis and DynamoDB streams even if you’re not hitting the throttling limits.
这些预测不应以票面价值为依据。 对于初学者来说,关于吞吐量和消息大小完全一致的假设是不现实的,并且即使您未达到节流限制,您仍需要Kinesis和DynamoDB流有一定的余量。
That said, what these projections do tell me is that:
也就是说,这些预测告诉我的是:
- You get an awful lot with each shard in Kinesis streamsKinesis流中的每个分片都会给您带来很多麻烦
- While there’s a baseline cost for using Kinesis streams, the cost goes down when usage scales up as compared to SNS and DynamoDB streams, thanks to the significantly lower cost per million requests尽管使用Kinesis流有基准成本,但与SNS和DynamoDB流相比,使用量增加时成本降低,这是因为每百万个请求的成本大大降低
Whilst SNS, Kinesis, and DynamoDB streams are your basic choices for the broker, Lambda functions can also act as brokers in their own right and propagate events to other services.
尽管SNS,Kinesis和DynamoDB流是代理的基本选择,但Lambda函数也可以自己充当代理并将事件传播到其他服务。
This is the approach used by the aws-lambda-fanout project from awslabs. It allows you to propagate events from Kinesis and DynamoDB streams to other services that cannot directly subscribe to the three basic choice of brokers (either because of account/region limitations or that they’re just not supported).
这是awslabs的aws-lambda-fanout项目使用的方法。 它允许您将事件从Kinesis和DynamoDB流传播到其他不能直接订阅代理的三种基本选择的服务(由于帐户/区域限制或不支持它们)。
While it’s a nice idea and definitely meets some specific needs, it’s worth bearing in mind the extra complexities it introduces, such as handling partial failures, dealing with downstream outages, misconfigurations, and so on.
尽管这是一个不错的主意,并且肯定可以满足某些特定需求,但值得记住的是它引入了额外的复杂性,例如处理部分故障,处理下游中断,配置错误等。
结论 (Conclusion)
So what is the best event source for doing pub-sub with AWS Lambda? Like most tech decisions, it depends on the problem you’re trying to solve, and the constraints you’re working with.
那么,使用AWS Lambda进行发布订阅的最佳事件源是什么? 像大多数高科技的决定,这取决于你试图解决的问题 ,而你正在使用的限制 。
In this post, we looked at SNS, Kinesis Streams, and DynamoDB Streams as candidates for the broker role. We walked through a number of scenarios to see how the choice of event source affects scalability, parallelism, and resilience against temporal issues and cost.
在这篇文章中,我们研究了SNS,Kinesis Streams和DynamoDB Streams作为代理角色的候选人。 我们遍历了许多场景,以了解事件源的选择如何影响可伸缩性,并行性以及针对时间问题和成本的弹性。
You should now have a much better understanding of the tradeoffs between various event sources when working with Lambda.
现在,使用Lambda时,您应该对各种事件源之间的权衡有了更好的了解。
Until next time!
直到下一次!
翻译自: https://www.freecodecamp.org/news/how-to-choose-the-best-event-source-for-pub-sub-messaging-with-aws-lambda-31ca4db9be69/
aws lambda使用
aws lambda使用_如何使用AWS Lambda为发布/订阅消息选择最佳事件源相关推荐
- aws lambda使用_如何使用AWS Lambda和S3构建无服务器URL缩短器
aws lambda使用 by Daniel Ireson 丹尼尔·埃里森(Daniel Ireson) 如何使用AWS Lambda和S3构建无服务器URL缩短器 (How to build a S ...
- aws 静态网站_如何使用AWS托管静态网站-入门指南
aws 静态网站 When I created my first portfolio last year, I based it on what I had learned from freeCode ...
- java lambda 反射_反射调用与Lambda表达式调用
想调用一个方法很容易,直接代码调用就行,这人人都会.其次呢,还可以使用反射.不过通过反射调用的性能会远远低于直接调用--至少从绝对时间上来看的确是这样.虽然这是个众所周知的现象,我们还是来写个程序来验 ...
- mysql lambda查询_从NodeJS AWS Lambda函数查询MySQL数据库
我在AWS Lambda函数中查询我的 MySQL数据库(从AWS远程托管)时遇到问题. 这是我的代码,除了Lambda函数的其余部分所需的部分(正在为Alexa技能调用): var mysql = ...
- java lambda 变量_为什么Java中lambda表达式不能改变外部变量的值,也不能定义自己的同名的本地变量呢?...
你问的问题在 Project Lambda 的概述文档上已经解释了,这都属于设计上的取舍. 不能改变外部变量的值是因为线程安全问题.当然这可能不是唯一原因,可能有其他考虑,但文档上清清楚楚说明了:Wh ...
- java lambda循环_使用Java 8 Lambda简化嵌套循环
java lambda循环 对于每个经常需要在Java 8(或更高版本)中使用多维数组的人来说,这只是一个快速技巧. 在这种情况下,您可能经常会以类似于以下代码的结尾: float[][] value ...
- java lambda循环_在Java 8 Lambda中创建自己的循环结构
java lambda循环 Java没有简单的结构可以重复N次. 当然,我们可以创建一个for循环,但是很多时候我们甚至都不关心在循环中创建的变量. 我们只想重复一些代码N次,仅此而已. 使用Java ...
- java lambda使用_在Java 8 Lambda上使用Apache Commons Functor功能接口
java lambda使用 Apache Commons Functor (以下称为[functor])是一个Apache Commons组件,它提供功能性的编程API和已实现的几种模式(访问者,生成 ...
- java lambda表达式_「JAVA8」- Lambda 表达式
Lambda 表达式,也可称为闭包,它是推动 Java 8 发布的最重要新特性. Lambda 允许把函数作为一个方法的参数(函数作为参数传递进方法中). 使用 Lambda 表达式可以使代码变的更加 ...
最新文章
- DescriptionAttribute Class
- Linux之CentOS找不到configure
- 领航服务器系统,应用领航:盘点那些年我们一起追过的OS
- leetCode 204. Count Primes 哈希 求素数
- websocket的加密和解密过程
- Map对象与实体类Object对象转换
- shell 学习之case语句
- 虚拟机外接USB设备情况的vMotion问题
- Ruckus R500 AP设置
- OpenCasCade——给定B样条曲线上的一点,求出过该点的切向量或法向量
- 大端与小端的区别 之小端
- Debain查看ip地址
- unity之摇杆和NPC
- 360打开html乱码怎么办,360浏览器出现乱码怎么回事_360浏览器页面乱码如何解决-win7之家...
- 阿里云国际站代理商:利用RDS MySQL数据库云开发ToDo List
- RFID固定资产管理系统,提高工作效率,节省时间-新导智能
- 4. PCIe 接口时序
- a55计算机主板,A55架构简介与A55主板赏析
- ffmpeg转码php配置,PHP+ffmpeg+nginx的配置实现视频转码
- android 仿微信通知栏