巨石加密

by Alan Ridlehoover

通过艾伦·里德尔霍弗

点餐：如何吃一个可怕的巨石 (Ordering Take Out: How to Eat a Scary Monolith)

Martin Fowler said:

马丁·福勒(Martin Fowler) 说 ：

Almost all the successful microservice stories have started with a monolith that got too big and was broken up.

几乎所有成功的微服务故事都是从一个庞大的整体开始的，而整体则被分解了。

You can add our story to that list.

您可以将我们的故事添加到该列表中。

My team and I just built a new messaging service. It replaced an aging subsystem in our primary application. We modeled the new service on real-life concepts. This simplified the data model considerably, unlocking new features that would have been a challenge to build in with the old data model.

我和我的团队刚刚建立了新的消息传递服务。 它取代了我们主要应用程序中的老化子系统。 我们根据现实生活中的概念对新服务进行了建模。 这极大地简化了数据模型，从而释放了新功能，而这些新功能将是旧数据模型内置的挑战。

Along the way, we learned a few things which we’d like to share…

一路上，我们学到了一些我们想分享的东西...

我们最坏的情况，最悲观的估计非常乐观 (Our worst-case, most pessimistic estimate was wildly optimistic)

Our most conservative guess for how long it would take to complete the project was three months. It took ten.

我们对完成该项目需要多长时间的最保守的猜测是三个月。 花了十点。

To be fair, we estimated that the entire team would take three months to complete the work. But, in reality, we took on several projects at once. We continued ongoing maintenance of the legacy subsystem, and even added new features. So, given less than half the resources, the original estimate should have been longer.

公平地说，我们估计整个团队将需要三个月才能完成工作。 但是，实际上，我们同时进行了多个项目。 我们继续对旧版子系统进行维护，甚至添加了新功能。 因此，在资源不到一半的情况下，原始估计值应该更长。

That said, we did underestimate the work. We underestimated the work related to disentangling the subsystem from the monolith. We started out searching for references to classes owned by the subsystem. We ended up looking at SQL queries that referenced the underlying database tables. That’s how tightly coupled the subsystem was to the rest of the application.

也就是说，我们确实低估了这项工作。 我们低估了与将子系统从整体中解开有关的工作。 我们开始搜索对子系统拥有的类的引用。 我们最终查看了引用基础数据库表SQL查询。 这就是子系统与应用程序其余部分紧密耦合的方式。

Lesson learned: Pay attention to all the integration points. There are likely more than you think.

经验教训：注意所有集成点。 可能超出您的想象。

围绕整体中的子系统构建一条护城河 (Build a moat around the subsystem in the monolith)

In Domain Driven Design, Eric Evans describes a pattern called the Anti-Corruption Layer. This layer is a wall between two subsystems. Neither subsystem talks to the other directly. They both talk to the wall. Following this pattern prevents the subsystems from bleeding into one another.

在域驱动设计中，埃里克·埃文斯( Eric Evans)描述了一种称为反腐败层的模式。 该层是两个子系统之间的墙。 两个子系统都没有直接与对方对话。 他们俩都在墙上说话。 遵循此模式可防止子系统渗入彼此。

For us, other subsystems had metastasized into ours. We needed a way of disconnecting them without modifying their functionality. We looked to the Anti-Corruption Layer for inspiration.

对于我们来说，其他子系统已经转移到我们的子系统中。 我们需要一种在不修改其功能的情况下将其断开连接的方法。 我们从反腐败层寻求灵感。

Our solution: we wrapped a façade layer around our subsystem within the monolith. This placed a moat around the subsystem, preventing access except through defined interfaces. It also allowed us to integration test the calls to the façade. We kept those tests passing throughout, even after cutting over to the new service.

我们的解决方案：我们在整体中的子系统周围包裹了立面层。 这在子系统周围造成了麻烦，阻止通过已定义的接口访问。 它还使我们能够对外观的调用进行集成测试。 即使在过渡到新服务之后，我们仍使这些测试始终通过。

As mentioned above, this was by far the most difficult piece of extracting the service. Pulling half of a raw SQL query behind a façade is hard work:

如上所述，这是迄今为止提取服务最困难的部分。 将一半的原始SQL查询拖到立面后面是艰巨的工作：

First, you have to understand the entire query, some of which were hundreds of lines long. Next, you extract the bits of the query that access the old subsystem. And, then, you convert them into the new data model. You extend the new service to support this new query. Finally, you integrate the results with the remainder of the original SQL query. If you do it right, the calling code never knows the difference.

首先，您必须了解整个查询，其中一些长达数百行。 接下来，提取访问旧子系统的查询的位。 然后，将它们转换为新的数据模型。 您扩展了新服务以支持此新查询。 最后，将结果与原始SQL查询的其余部分集成在一起。 如果操作正确，则调用代码永远不会知道区别。

Lesson learned: When constructing an application, it is important to maintain boundaries. Encapsulation is important. That point is obvious outside the monolith. But, it is even more important inside the monolith. It’s the only way to keep the system pliable, so you can make room for new features through refactoring.

经验教训：构建应用程序时，保持界限很重要。 封装很重要。 在整体之外，这一点是显而易见的。 但是，在整体内部更重要。 这是保持系统柔韧性的唯一方法，因此您可以通过重构为新功能腾出空间。

并行运行两个系统 (Run both systems in parallel)

We ran both systems in parallel for months while we worked toward feature parity. We could do this because we created the façade mentioned above. We sent requests to the façade through both the legacy subsystem and the new service. Comparing the results enabled us to find (and fix) data inconsistencies. It also gave us a real-world stress test against live production data.

在努力实现功能均等的同时，我们并行运行了两个系统几个月。 我们可以这样做是因为我们创建了上面提到的外观。 我们通过旧版子系统和新服务向立面发送了请求。 比较结果使我们能够发现(并修复)数据不一致。 它还为我们提供了针对实时生产数据的真实压力测试。

Lesson learned: We found most of our bugs this way. There’s no substitute for production workloads. We admit that this increased development costs. But, finding the bugs before we put the system in production was worth every penny.

获得的经验：我们通过这种方式发现了大多数错误。 生产工作量无可替代。 我们承认，这增加了开发成本。 但是，在我们将系统投入生产之前发现错误的每一分钱都是值得的。

使用功能标志来分阶段发布 (Use feature flags to stage rollout)

A feature flag is a boolean value that enables a feature when on, and disables the feature when turned off. Typical feature flag systems allow you to turn the flag on for subsets of your users. This allows you to roll out functionality gradually, rather than all at once.

功能标志是一个布尔值，可在启用时启用功能，在关闭时禁用功能。 典型的功能标志系统允许您为用户的子集打开标志。 这使您可以逐步推出功能，而不是一次全部推出。

Our implementation used four separate flags:

我们的实现使用了四个单独的标志：

• One flag controlled whether to write to the new system. This allowed us to run the systems in parallel. We turned this flag on very early in the process, giving us the benefits described above.一个标志控制是否写入新系统。 这使我们可以并行运行系统。 我们在此过程的很早就启用了此标志，从而为我们带来了上述好处。
• Another flag controlled whether to read data from the new service. This way we could test the functionality before enabling it globally.另一个标志控制是否从新服务读取数据。 这样，我们可以在全局启用功能之前对其进行测试。
• The last two flags controlled which system could access our third-party email providers. One flag turned off the legacy subsystem. The other turned on the new service. We separated these flags so we could turn both systems off at the same time, in the event of a major problem. (We did not end up exercising this functionality. But we’re still glad we built it.)最后两个标志控制哪个系统可以访问我们的第三方电子邮件提供商。 一个标志关闭了旧子系统。 另一个打开了新服务。 我们分离了这些标志，以便在出现重大问题时可以同时关闭两个系统。 (我们最终没有使用此功能。但是，我们仍然很高兴我们构建了它。)

Finally, using feature flags allowed us to roll out the new service to one customer at a time. This reduced risk and prevented undue disruptions to the business.

最后，使用功能标志使我们能够一次向一位客户推出新服务。 这样可以降低风险并防止对业务造成不必要的中断。

Lesson learned: Use separate feature flags for writing to and reading from a service. This allows you to begin running the service in parallel with the legacy system. And, in our case, adding the extra flags to prevent duplicate emails from being sent was critical.

经验教训：使用单独的功能标志进行服务的写入和读取。 这使您可以开始与旧系统并行运行服务。 而且，在我们的案例中，添加额外的标志以防止发送重复的电子邮件至关重要。

尽早添加日志记录，异常处理和监视 (Add logging, exception handling & monitoring early)

One of the first things we built inside the new service was a logger. Next, we added exception handling. This turned out to be crucial when debugging the system. We found it especially useful across the service boundary.

我们在新服务中构建的第一件事就是记录器。 接下来，我们添加了异常处理。 事实证明，这在调试系统时至关重要。 我们发现跨服务边界特别有用。

We added monitoring later to give us a window into our data import process. Adding it was easy because of our centralized logger.

稍后我们添加了监视，以便为我们提供一个进入数据导入过程的窗口。 由于我们的集中式记录器，添加起来很容易。

Lesson learned: Centralize your logging and exception handling, and build it early. You’ll thank yourself later.

经验教训：集中您的日志记录和异常处理，并尽早构建它。 稍后您会感谢您的。

迁移数据时会遇到问题 (Expect issues migrating your data)

We decided to port our data into the new service, so there would be one system of record. We did this by sending messages to the new service, thus testing the interface under heavy load.

我们决定将数据移植到新服务中，因此会有一个记录系统。 为此，我们将消息发送到新服务，从而在高负载下测试了接口。

The vast majority of the data migrated correctly. But there were many valid edge cases. Each time we would figure out one of the edge cases, we would adjust the migration and run it again.

绝大多数数据已正确迁移。 但是有许多有效的边缘情况。 每次我们找出其中一种情况时，都会调整迁移并再次运行。

In the end, we had about 700 records (out of 1.2M) left over that we could not explain. The vast majority were several years old. This data did not fit any of the corner cases we’d identified and resolved. After spending a couple of days on it, we decided to mark the records as “failed” and move on.

最终，我们还有大约700条记录(120万条记录)还无法解释。 绝大多数是几岁。 此数据不适合我们确定和解决的任何极端情况。 花了几天时间后，我们决定将记录标记为“失败”并继续前进。

Lesson learned: Production data is sloppy. Most of it will look correct. Some of it will not. The older the data, the harder it is to migrate. Records will be missing. Foreign keys won’t exist. Just roll with it. Make the best, most user friendly decisions you can.

经验教训：生产数据草率。 大多数看起来正确。 其中一些不会。 数据越旧，迁移就越困难。 记录将丢失。 外键将不存在。 随它滚动。 做出最佳，最人性化的决定。

使用确定性UUID和幂等迁移 (Use deterministic UUIDs and idempotent migrations)

When you have two different systems running in parallel, you need a shared id space. Any UUID will do. But we chose to use deterministic UUIDs to support idempotent migrations. To generate a deterministic UUID, you provide a namespace and some unique attribute. Given the same data, the algorithm always produces the same UUID.

当两个不同的系统并行运行时，需要共享的ID空间。 任何UUID都可以。 但是我们选择使用确定性UUID来支持幂等迁移。 要生成确定性的UUID，请提供名称空间和一些唯一的属性。 给定相同的数据，该算法始终会产生相同的UUID。

In our case, the façade in the main application generates the deterministic UUIDs. Then it sends the UUID to the service for storage. Inside the service, we indexed that field to prevent duplicates. This makes idempotency possible.

在我们的情况下，主应用程序中的外观会生成确定性的UUID。 然后，它将UUID发送到服务进行存储。 在服务内部，我们对该字段建立了索引，以防止重复。 这使得幂等成为可能。

Lesson learned: Your production data will be unpredictable. As you resolve corner cases, you will need to run your migrations over and over. Using deterministic UUIDs makes this possible.

获得的经验：您的生产数据将不可预测。 解决极端情况时，将需要一遍又一遍地运行迁移。 使用确定性的UUID使这成为可能。

使用队列 (Use queues)

Introducing a new service means introducing another point of failure. We wanted to protect ourselves from minor service outages. So, we decided to communicate with the service via a queue. By using a robust queue, we achieved a bit of fault tolerance should the service ever go down. We also chose a FIFO queue to guarantee the order of operations. This prevents an update from preceding a create, or following a delete. This also allowed us to speed up our data import process by scaling horizontally.

引入新服务意味着引入另一个故障点。 我们希望保护自己免受轻微服务中断的影响。 因此，我们决定通过队列与服务进行通信。 通过使用健壮的队列，我们​​可以在服务中断时达到一定程度的容错能力。 我们还选择了FIFO队列来保证操作顺序。 这样可以防止在创建之前或删除之后进行更新。 这也使我们可以通过水平缩放来加快数据导入过程。

Lesson learned: There are several advantages to using a robust queue. Fault tolerance lends peace of mind. And, horizontal scaling enables you to keep up with demand.

经验教训：使用健壮的队列有许多优点。 容错使您安心。 而且，水平缩放使您能够满足需求。

使用断路器 (Use circuit breakers)

Reading data asynchronously via a queue requires the caller to understand asynchronous callbacks. JavaScript is, of course, quite good at this. But, Ruby is not. And, in this case, it is the monolith calling the service via a queue.

通过队列异步读取数据需要调用方了解异步回调。 当然，JavaScript非常擅长于此。 但是，Ruby不是。 并且，在这种情况下，是通过队列调用服务的整体。

Consider a request from the front-end to fetch some data. The monolith would receive that request and place a message on a queue along with a correlation id. The service would then reply (on a different queue) with the results and the same correlation id. But, the worker that processes the response would not have a handle on the request. So, now you’d need to push the data to the front-end (most likely using sockets).

考虑来自前端的请求以获取一些数据。 整体将接收该请求并将消息以及相关性ID放置在队列中。 然后，该服务将(在不同的队列上)使用结果和相同的关联ID进行回复。 但是，处理响应的工作人员将无法处理请求。 因此，现在您需要将数据推送到前端(最有可能使用套接字)。

In other words, we would have had to rewrite our front-end to receive data via sockets. f. Unfortunately, we did not have time to rebuild the front-end of our application to work this way. So, we chose to use HTTP for read operations. This worked well, until it didn’t.

换句话说，我们必须重写前端才能通过套接字接收数据。 F。 不幸的是，我们没有时间重建应用程序的前端以这种方式工作。 因此，我们选择使用HTTP进行读取操作。 这很好，直到没有。

During testing, a bug in the deployment process took the new service down. This prevented the monolith from loading in our staging environment. Why? Because we were bootstrapping data from the service when we loaded the first page. Since the service was down, the requests were all timing out. The solution was to use the circuit breaker pattern.

在测试期间，部署过程中的错误使新服务中断。 这阻止了整体加载到我们的暂存环境中。 为什么？ 因为我们在加载第一页时从服务引导数据。 由于服务已关闭，因此所有请求均已超时。 解决方案是使用断路器模式。

Circuit breakers wrap calls to external services. If the call works, nothing happens. But if the call fails with certain exceptions (like a time out or a server error), the circuit breaker trips. While tripped, the circuit breaker won’t call the service. Instead, it returns a hard-coded return value like `nil` or `[]` or `{}` during a cool down period. Once the cool down expires, the circuit breaker starts calling the service again. If it’s back up, great! If not, the circuit breaker trips again.

断路器包装对外部服务的呼叫。 如果该呼叫有效，则什么都不会发生。 但是，如果呼叫由于某些异常(例如超时或服务器错误)而失败，则断路器将跳闸。 跳闸时，断路器不会致电服务人员。 相反，它在冷却期间返回硬编码的返回值，例如`nil`或`[]`或`{}`。 一旦冷却时间结束，断路器将再次开始呼叫服务。 如果备份了，那就太好了！ 如果不是，则断路器再次跳闸。

Lesson learned: If you must use HTTP, protect the broader system from service outages. Circuit breakers are one mechanism for doing this.

经验教训：如果必须使用HTTP，请保护更广泛的系统免受服务中断的影响。 断路器是执行此操作的一种机制。

行政赞助 (Executive Sponsorship)

The success of long term projects often comes down to executive sponsorship. We were fortunate that both our VP of Product and our VP of Engineering were on board with our plans. They put a great deal of trust in us. And when the project began to look bigger than we’d planned, they stood by us. We could not have completed our work without their support.

长期项目的成功通常取决于高层管理人员的赞助。 我们很幸运，我们的产品副总裁和工程副总裁都参与了我们的计划。 他们对我们非常信任。 当项目看起来比我们计划的要大时，他们就站在我们身边。 没有他们的支持，我们不可能完成我们的工作。

Why were our executive sponsors willing to go to bat for us? Several reasons. But one of the most important is that we were open and honest in our communications with them. They knew where the project was and they knew what we needed to do.

为什么我们的执行发起人愿意为我们踢球？ 几个原因。 但最重要的一点是，我们在与他们的沟通中保持开放和诚实。 他们知道项目在哪里，也知道我们需要做什么。

Lesson learned: When embarking on a long-term, high risk project, make sure you have the support of your leadership. Make sure they understand the benefits as well as the risks. And, communicate continuously. It builds trust, which you’ll need if you’re going to succeed.

经验教训：在着手进行长期的高风险项目时，请确保获得领导层的支持。 确保他们了解收益和风险。 并且，不断沟通。 它建立了信任，如果您要成功，就需要信任。

我们会做些什么？ (What would we do differently?)

We thought we did our due diligence before proposing the project. We found every reference to the models used in the subsystem. But we did not search for direct references to the underlying table names in raw SQL statements. That was an oversight. We will definitely do that next time.

我们以为在提出该项目之前已经进行了尽职调查。 我们找到了对子系统中使用的模型的所有引用。 但是我们没有在原始SQL语句中搜索对基础表名称的直接引用。 那是一个疏忽。 下次我们一定会这样做。

If we could go back in time, we would isolate the old subsystem within the monolith. Encapsulating the subsystem would simplify the extraction process. In fact, it may even have made it unnecessary, since we could have be able to refactor in place.

如果我们可以回到过去，则可以将整体子系统中的旧子系统隔离开。 封装子系统将简化提取过程。 实际上，它甚至可能使它变得不必要，因为我们可以原地重构。

那么，我们会再做一次吗？ (So, would we do it again?)

It was a long journey, for sure. Extricating our subsystem from the monolith was more difficult that we predicted due to some truly epic coupling. We did the decoupling work out of necessity. It was not enjoyable. It felt like a chore.

当然，这是一段漫长的旅程。 由于某些真正的史诗般的耦合，从整体中解脱我们的子系统比我们预测的要困难得多。 我们出于必要进行了去耦工作。 这是不愉快的。 感觉像家务。

But, working with the new service is a dream. The code is very well factored. We have super high test coverage. And, we have well defined domain logic that corresponds to specific use cases. So, we can tell what the application does by looking at a single class. This means that extending the service is as simple as adding another domain class and tests for it.

但是，使用新服务是一个梦想。 该代码是非常好的因素。 我们拥有超高的测试覆盖率。 并且，我们有定义明确的域逻辑，对应于特定的用例。 因此，我们可以通过查看单个类来判断应用程序的功能。 这意味着扩展服务就像添加另一个域类并对其进行测试一样简单。

But, would we do it again?

但是，我们会再做一次吗？

As engineers? We would definitely do it again. The project led to large improvements in developer productivity and happiness. The trade-off was worth it for us in job satisfaction alone. In fact, we already see opportunities for extracting several small services.

作为工程师？ 我们一定会再做一次。 该项目大大提高了开发人员的生产力和幸福感。 仅在工作满意度方面，权衡对我们来说是值得的。 实际上，我们已经看到了提取几个小型服务的机会。

As an organization? We are very happy with the results we achieved. Customers are happy we’re shipping long-requested features we were unable to ship before. Management is happy that there have been no production issues. And, the team is thrilled to be free of the monolith.

作为组织？ 我们对取得的成果感到非常满意。 客户很高兴我们交付了我们以前无法交付的长期要求的功能。 管理层很高兴没有生产问题。 而且，该团队很高兴摆脱了巨石。

That said, this project was a significant investment for us at our stage. We will need to see a strong return before taking on another large scale bet on service extractions.

就是说，这个项目对我们现阶段来说是一笔巨大的投资。 在对服务提取进行另一次大规模押注之前，我们将需要看到强劲的回报。

A version of this article was first published on SourceCode, our blog about engineering in the recruiting industry.

本文的一个版本首先发布在SourceCode上 ，即我们关于招聘行业工程的博客。

I’d like to thank Fito von Zastrow, Jason Rosendale, Ryan A Booth, and Cole Goeppinger, all of whom made invaluable contributions to this article.

我要感谢Fito von Zastrow ， Jason Rosendale ， Ryan A Booth和Cole Goeppinger ，他们全都为本文做出了宝贵的贡献。

And, thank you! You win a prize for reading this far. Mention this article to me IRL for a free sticker!

而且，谢谢！ 您将获得阅读到目前为止的大奖。 向我提及这篇文章，可以免费获得IRL！

翻译自: https://www.freecodecamp.org/news/ordering-take-out-how-to-eat-a-scary-monolith-805e471a613f/

巨石加密