Modularizing the monolith: a real-world experience

Microservices have lots of benefits (and downsides), but one common reason people move away from monoliths is that the project became a big ball of mud, and the thinking that cutting it into several services will fix it (it probably won’t — you’ll have multiple smaller balls of mud).

A nice middle-ground is modular monoliths. Well, what does that mean? It means that you divide your monolith into several mini-apps. I might use the word “Domain”, from DDD, erroneously at this article sometimes.

It is common for us to group things by Type. For instance, this would be a common structure for a Laravel project:

 1├── Actions
 2│   ├── RefundPayment.php
 3│   ├── CancelCustomerSubscription.php
 4│   ├── DeleteSite.php
 5├── QueryBuilders
 6│   └── SiteQueryBuilder.php
 7├── Console
 8│   ├── Commands
 9│   └── Kernel.php
10├── Contracts
11│   ├── HasSites.php
12│   ├── MakesHttpRequests.php
13├── Enums
14│   ├── SiteType.php
15│   ├── PaymentStatus.php
16│   └── UserRole.php
17├── Events
18│   ├── CustomerCanceledSubscription.php
19│   └── CustomerMadePayment.php
20│   └── SiteCreated.php
21├── Exceptions
22│   ├── CouldNotUpdateTwilioDetailsException.php
23│   └── Handler.php
24├── Facades
25│   └── Twilio.php
26├── Http
27│   ├── Controllers
28│   │   ├── Api
29│   │   │   ├── SiteController.php
30│   │   ├── Controller.php
31│   │   ├── DashboardController.php
32│   │   ├── InvoiceController.php
33│   ├── Kernel.php
34│   ├── Livewire
35│   ├── Middleware
36│   ├── Requests
37│   │   ├── StartTrialRequest.php
38│   │   └── StoreSiteRequest.php
39│   ├── Resources
40│   │   ├── UserResource.php
41│   │   ├── SiteResource.php
42│   │   ├── InvoiceResource.php
43├── Jobs
44│   ├── UpdateSiteStats.php
45│   ├── RefreshInvoice.php
46│   ├── UnlockTwilioAccounts.php
47├── Listeners
48│   ├── AddCustomerToCustomersEmailList.php
49│   ├── AddCustomerToTrialEmailList.php
50│   ├── RemoveCustomerFromMembersLists.php
51│   ├── UpdateSiteDnsRecords.php
52│   ├── SendPaymentMadeEmail.php
53│   ├── SendSubscriptionCanceledEmail.php
54├── Mail
55│   ├── PaymentFailedEmail.php
56│   ├── PaymentMadeEmail.php
57│   ├── RebillNotice.php
58├── Models
59│   ├── Customer.php
60│   ├── Invoice.php
61│   ├── Payment.php
62├── Notifications
63├── Observers
64│   ├── SiteObserver.php
65├── Policies
66├── Providers
67├── Rules
68├── Services
69├── Support
70├── Traits
71│   ├── HasLicenses.php
72│   └── MemoizesValues.php

I actually simplified this a lot, but you get the idea — as the app grows, this starts to get very confusing. For instance, what do UpdateSiteStats and RefreshInvoice have in common? Nothing, except that they’re both queued jobs — they have the same Type. That’s the reason they’re on the same folder — but they don’t share a lot. In fact, they don’t share anything — they belong to two different modules.

With that said, is this organization bad? Not at all. This will work for many — I’d say most — applications, but it’s possible it’ll make things too confusing if your application is too dense. That’s where modular monoliths come into play.

A simple way to think of it is that you split your domains — or “modules” — into several mini-applications and (may) implement boundaries between them. Keep in mind this is not a domain-driven design article — it’s simply about splitting your application.

For example, you might have a Billing domain that deals with payments, invoices, etc., and also a Sites domain, that deals with custom sites for the clients using the service. Those two might communicate sometimes (for instance, your application might calculate the monthly bill based on the amount of Sites a customer has), but overall they’re pretty independent. Here’s what it could look like based on the example I’ve shown earlier:

 1Modules/
 2├── Billing/
 3│   ├── Actions/
 4│   │   └── RefundPayment.php
 5│   ├── Models/
 6│   │   ├── Invoice.php
 7│   │   └── Payment.php
 8│   ├── Enums/
 9│   │   └── PaymentStatus.php
10│   ├── Events/
11│   │   └── PaymentMade.php
12│   ├── Exceptions/
13│   │   └── PaymentFailedException.php
14│   └── Jobs/
15│       └── RefreshInvoice.php
16└── Sites/
17    ├── Actions/
18    │   └── DeleteSite.php
19    ├── Models/
20    │   └── Site.php
21    ├── Enums/
22    │   └── SiteType.php
23    ├── Events/
24    │   └── SiteCreated.php
25    └── Jobs/
26        └── UpdateSiteStatus.php

This is a rather simple example but you can see that now things are grouped by Domain -> Type. That should make things easier, right?
Well, maybe not so much. Where are the providers, controllers, API resources, mailables, middlewares…? Should migrations still be tangled together with each other in the database/migrations folder? What about the views? Do they stay all together?

Okay, so let’s take a break and think about a couple of points.

• Laravel (and most frameworks) allow you to organize your apps however you want. You could keep the “domain stuff” inside the Modules folder and the “application stuff” where it usually stays. Or maybe you can treat each module as a mini-application that includes everything — migrations, controllers, API resources, tests, your credit card details — or you can be a bit more conservative and leave some things out. It is up to you and there is no right answer.
• This is a very progressive approach. You don’t have to do it all at once — you can move things in small increments, in a way that doesn’t harm your team’s productivity or feature delivery speed.
• As I said, you’re free to organize things as you want. Some people will try to force you that there’s a correct way of doing things, but I don’t think there is. You can follow a DDD philosophy and take inspiration from other well-defined architectures, but it is very unlikely that you’ll be able to follow anything by the book, especially when working in existing, large projects.

Before we move on, I suggest you add these two articles from Shopify engineering blog to your reading list:
Deconstructing the Monolith
Under Deconstruction: The State of Shopify’s Monolith — Development

Spoiler: they made several mini Rails applications inside their main monolithic application.

Identifying the problem

In the case of the application I was working on, the problem was there was a huge cognitive load to work on a single feature. When I was working on Billing, I didn’t really care about Sites or Contacts. At least not directly.
Besides that, navigating the app was also a bit hard since everything was tangled together. Lots of functionalities were coupled together so maintaining them was a bit harder.

We’re using Eloquent here, so I never expected to decouple features completely — domains are always going to leak through Eloquent models.

The goal was to limit what needed to be touched whenever working on a user story. Is this a Billing feature? Okay, cool, I want to see billing-stuff only. Sure, maybe I’ll have to take a plane to Sitesland to calculate how much a user should be charged, but that’s a quick trip.

Tackling down progressively

I did not want to spend a lot of time moving things around, especially because, well, we needed to deliver and fix features.
I started to slowly group things by domain, not caring much about the application layer.
This is not especially hard — it’s just moving things around and changing namespaces — the problem comes down to coupling. More often than not some components depended and maybe were even used by more than one Thing. Some methods had weird signatures, arrays and destructured data were being passed around, etc.

I want to add one more thing: we had a robust test suite, so that helped make things easier immensely.

I ended up making weekly pull requests to “extract” those modules out of the big ball of mud. I was looking for a couple of things:

• To extract the domain logic
• To refactor things to accept DTOs instead of mysterious data or Eloquent models
• Making sure tests pass after all this

That didn’t give me well-defined boundaries — communication with the extracted module was better, yes — but that module would still communicate with the big ball of mud. So it’s something progressive — tough to do at once unless you halt any development.

At this point, the app structure would’ve looked like this:

 1app/
 2├── Actions/
 3│   ├── CancelCustomerSubscription.php
 4│   └── …
 5├── Console/
 6│   └── …
 7├── Contracts
 8├── Enums/
 9│   ├── SiteType.php
10│   └── …
11├── Events
12├── Exceptions
13├── Http/
14│   ├── Controllers/
15│   │   ├── InvoiceController.php
16│   │   └── …
17│   ├── Middleware
18│   ├── Requests
19│   └── Resources/
20│       ├── InvoiceResource.php
21│       └── …
22└── …
23Modules/
24└── Billing/
25    ├── Actions/
26    │   ├── RefundPayment.php
27    │   └── CreateInvoice.php
28    ├── Models/
29    │   ├── Invoice.php
30    │   ├── InvoiceItem.php
31    │   └── Payment.php
32    ├── Contracts/
33    │   └── BillableContract.php
34    ├── Enums/
35    │   └── PaymentStatus.php
36    ├── Events/
37    │   └── PaymentMade.php
38    ├── Exceptions
39    ├── Jobs
40    └── DataTransferObjects/
41        ├── PendingInvoiceDto.php
42        ├── InvoiceDto.php
43        └── PaymentDto.php

So you can see that only the Domain of the Billing module was extracted.
Could I have extracted the application layer? Yes, but that would’ve made the pull request bigger and more error-prone. Small and progressive was what I was looking for.

You’ll notice that a couple of DTOs were introduced — that made interacting with the Billing domain a bit easier. For instance, a job that calculates , makes a payment and generates an invoice could look like this:

 1<?php
 2 
 3namespace App\Jobs;
 4 
 5class ChargeUser
 6{
 7    public function __construct(
 8        protected CreateInvoice $createInvoiceAction,
 9        protected ChargeUser $chargeUserAction
10    } {}
11 
12    public function handle(User $user)
13    {
14        $sites = $user->sites;
15 
16        $pendingInvoice = PendingInvoice::make(
17            userId: $user->id,
18            items: $sites
19        );
20 
21        try {
22            $payment = $this->chargeUserAction->handle($pendingInvoice);
23        } catch (PaymentFailedException $exception) {
24            return;
25        }
26 
27        $this->createInvoiceAction->handle($pendingInvoice, $payment);
28    }
29}
30 
31class PendingInvoice
32{
33    /**
34    /* @param int $userId
35    /* @param BillableContract[] $items
36    */
37    public function __construct(
38        public int $userId,
39        public array $items
40    ) {}
41 
42    public function total(): float
43    {
44        return array_reduce($this->items, fn (BillableContract $billable) => $billable->total());
45    }
46}
47 
48class ChargeUserAction
49{
50    public function __construct(
51        protected PendingInvoice $pendingInvoice,
52        protected PaymentProviderContract $paymentProvider
53    ) {}
54 
55    public function handle()
56    {
57        $providerPayment = $this->paymentProvider->charge($this->pendingInvoice->userId, $this->pendingInvoice->total());
58 
59        // some logic
60        return $payment;
61    }
62}
63 
64class CreateInvoiceAction
65{
66    public function __construct(
67        protected PendingInvoice $pendingInvoice,
68        protected Payment $payment
69    ) {}
70 
71    public function handle(): Invoice
72    {
73        $invoice = Invoice::create([
74            'total' => $pendingInvoice->total(),
75            'payment_id' => $this->payment->id,
76            'user_id' => $pendingInvoice->userId
77        ]);
78 
79        $invoice->attachItems($pendingInvoice->items);
80 
81        return $invoice;
82    }
83}

So as you can see you end up with small, composable actions and structured ways to pass data around. The modules are not isolated by any means, but things have their place.
For instance, User is not a part of the Billing module, and it is still being used — but everything else —, payments, invoices and the actions, are.
The actions still return Eloquent models, Eloquent relationships are still used and that’s okay. If you want to leverage Eloquent, you have to be okay with things like these.

As you can see, as you migrate each Thingto it’s Module (or Domain), you still have to refactor some things outside of it, maybe create some actions, but overall it isn’t an extremely hard process. The key thing here is that you can do this as you go — you’re not required, by any means, to rearchitect the entire application at once. In fact, you don’t even have to move the entire Domain Layer(like we did on the example) at once.

Moving the rest of the module

We basically moved the Domain Layer of Billing. We still have controllers, requests, API resources, etc. on the appfolder. and Migrations on the database/migrations folder.
Moving things such as Policies, Views Routing, and Migrations to other places is not as straightforward as moving the domain layer — that’s because you have to instruct Laravel from where it should load those from.

To do that we can use Service Providers. Let’s add one inside Modules/Billing/Providers/BillingServiceProvider.php

 1<?php
 2 
 3namespace Modules\Billing\Providers;
 4 
 5class BillingServiceProvider extends ServiceProvider
 6{
 7    public function boot()
 8    {
 9        $this->loadRoutesFrom(__DIR__ . '../Routes/web.php');
10        $this->loadMigrationsFrom(__DIR__ . '../Database/Migrations');
11        $this->loadViewsFrom(__DIR__ . '../Views', 'billing');
12    }
13}

That does a couple of things:

• Loads a route file from Modules/Billing/Routes/web.php
• Loads migrations from Modules/Billing/Database/Migrations
• Loads views from Modules/Billing/Viewsusing the billing namespace — which means you’d call the view app.blade.php as billing::app

If you take a look at the last module tree, you’ll see that the Billing folder contained, basically, domain code, but it now has other things.
Since we added a routes file, we might as well move controllers, API resources, requests, middleware, etc — the Application Layer. That’d leave us with a bunch of things and it’s starting to get confused once again. That’s a good time to make things even more specific — separate what is Domain, Infrastructure, and Application code. That gives us something like this:

 1Modules/
 2└── Billing/
 3    ├── Domain/
 4    │   ├── Actions/
 5    │   │   ├── RefundPayment.php
 6    │   │   └── CreateInvoice.php
 7    │   ├── Models/
 8    │   │   ├── Invoice.php
 9    │   │   ├── InvoiceItem.php
10    │   │   └── Payment.php
11    │   ├── Contracts/
12    │   │   └── BillableContract.php
13    │   ├── Enums/
14    │   │   └── PaymentStatus.php
15    │   ├── Events/
16    │   │   └── PaymentMade.php
17    │   ├── Exceptions
18    │   ├── Jobs
19    │   └── DataTransferObjects/
20    │       ├── PendingInvoiceDto.php
21    │       ├── InvoiceDto.php
22    │       └── PaymentDto.php
23    ├── Application/
24    │   ├── Http/
25    │   │   ├── Controllers/
26    │   │   │   └── InvoiceController.php
27    │   │   ├── Middleware
28    │   │   ├── Requests
29    │   │   └── Resources/
30    │   │       └── InvoiceResource.php
31    │   └── Views/
32    │       └── invoices/
33    │           └── index.blade.php
34    ├── Routes/
35    │   └── web.php
36    └── Database/
37        ├── Migrations
38        ├── Factories
39        └── Seeders

You could also add an Infrastructure directory, but for simplicity I only moved the largest ones. NOTE: This is not DDD: this is just us splitting an application into modules.

As you can see, things are now very well segregated. If you needed to work on the Billing piece of the application, you shouldn’t have to leave that base directory.

“Oh but I need to charge users for Contacts now”. What do I do?
Well, you can see that the jobs accept PendingInvoiceDto, which accepts an array of BillableContract . Just include those contacts in that array and that’s it — you don’t really have to touch any code inside Billing. Need to add some VAT calculation? Okay, you can do that inside Billing-World. No need to look at anything else.

Going further

We already moved a lot of things, and the app folder no longer contains anything directly related to Billing. The only missing piece is… tests. The Billing tests are still tangled with the rest of our application.
You’re not someone who follows the rules, and you want to have the module’s tests inside the module’s space. How can you do that? It is surprisingly simple. We can just add a Modules/Billing/tests directory and instruct PHPUnit to look there.

 1<?xml version="1.0" encoding="UTF-8"?>
 2<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 3         xsi:noNamespaceSchemaLocation="./vendor/phpunit/phpunit/phpunit.xsd"
 4         bootstrap="vendor/autoload.php"
 5         colors="true"
 6>
 7    <testsuites>
 8        <testsuite name="Unit">
 9            <directory suffix="Test.php">./tests/Unit</directory>
10        </testsuite>
11        <testsuite name="Feature">
12            <directory suffix="Test.php">./tests/Feature</directory>
13        </testsuite>
14        <testsuite name="Modules Unit"> 
15            <directory suffix="Test.php">./Modules/**/tests/Unit</directory> 
16        </testsuite> 
17        <testsuite name="Modules Feature"> 
18            <directory suffix="Test.php">./Modules/**/tests/Feature</directory> 
19        </testsuite> 
20    </testsuites>
21</phpunit>

Once you move everything over you can kill the root testsfolder and then delete it from phpunit.xml.