Modern web applications have become more complex over the years as user requirements continue to push for more performance, less latency, and more features. As a developer, choosing Laravel is easy. But when you get beyond the application platform, you have to be thinking about data.
Does the database fit my needs? Will it scale? Can I report from it? And most systems require a search feature. What if the database you chose was AWS DynamoDB? The Laravel ecosystem has libraries for using Eloquent to store data in DynamoDB. But since it's a simple key/value datastore, searching is limited to indexes and expensive filters.
This is where a purpose-built search database like Typesense really shines. It handles those tough searches, ranks, and scores the results, and abstracts that complexity into an API for a developer to use. But how would you populate this secondary store? Traditional approaches would say to use Scout. But in this article, I want to demonstrate a more AWS native way by using DynamoDB's Stream capabilities and AWS Lambda to keep our primary DynamoDB table in sync with a Typesense collection.
Disclosure
But before we begin and for disclosure, Typesense sponsored me to experiment with their product and report my findings. They have rented my attention, but not my opinion. Here is my unbiased view of my experience as a developer when integrating search into a Laravel Application that includes AWS DynamoDB and Typesense.
Architecture
Before diving into the code, let's have a look at a macro view
of our solution. Think of a very traditional Laravel application
which has some Blade views, Controllers, and a simple Todo model.
The user can list the Todos, create new ones, and then search on
the name
and description
. For storage, we
are going to use DynamoDB and Typesense's Cloud option.
Let's dive in!
The Build
Todo Model
Let's jump in with our base level Todo model. I find that working from the bottom up helps stitch things together until I've got the full picture like in the architecture diagram above.
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use BaoPham\DynamoDb\DynamoDbModel;
class Todo extends DynamoDbModel
{
use HasFactory;
protected $primaryKey = 'id'; // required
}
Notice that I'm extending my model from
DynamoDbModel
. This comes from the widely popular and
well-maintained library in this GitHub repository..
What's nice about this library is that I can use the same
Laravel Eloquent methods that I'm used to. In the
TodoController
, the save looks like it normally would
even though I'm using DynamoDB.
public function store(request $request): redirectresponse
{
$todo = new todo;
$todo->id = Uuid::uuid4()->toString();
$todo->name = $request->name;
$todo->description = $request->description;
$todo->save();
return redirect('/todos')->with('status', 'todo data has been inserted');
}
Typesense Cloud and Collection
Before looking at connecting DynamoDB to Typesense, I want to show you how easy and powerful using the Typesense Cloud offering is. You could 100% run Typesense yourself and if you are self-hosting or VM-hosting your Laravel app, that might be the approach you want to take. But if you are deploying on AWS and using DynamoDB, a Cloud-hosted Typesense makes a ton of sense.
What I really enjoy about the cloud offering is that I can manage the cluster at an abstracted level. I set things like memory, nodes, whether I want high-availability, and high disk performance and Typesense takes care of the rest. All that I need to do from there is generate some API keys, attach permissions, and my clients are free to perform the operations needed.
Cluster configuration ends up looking like the below image.
With the cluster in place, other Typesense operations are opened up. I must define a Collection, with a document schema, and then I can begin inserting documents. For instance, here's how searching our Todos Collection shows up.
Connecting DynamoDB to Typesense
With our Typesense Cloud Cluster established, let's talk about syncing with DynamoDB. When building systems with Cloud native technologies like DynamoDB, sometimes it makes more sense to stay in that ecosystem to do additional processing like syncing data with DynamoDB and Typesense.
If you aren't familiar with AWS, DynamoDB offers a set of capabilities to respond to changes in real time via Streams. A DynamoDB stream can be read from AWS Kinesis or a Lambda function. For this example, a Lambda function is the route I'm going to take. It's simple, isolated, and I don't need the extensibility that Kinesis provides.
DynamoDB offers the option to read changes in a few varieties. I can read just Key changes, New Images, or New and Old Images. I've chosen to just read New Images so that I get the full DynamoDB item vs having to go back and do a double read to hydrate the item later. For this implementation, I'm using Golang to read from the stream and populate Typesense. I find Golang to be a solid choice when doing event-driven work in AWS Lambda.
Without diving into how it all works (there's a repository link at the end), here are the Typesense specific bits. Another solid plus for Typesense is that there are plenty of options when working with their API from an SDK standpoint.
// New Client
client = typesense.NewClient(
typesense.WithServer(url),
typesense.WithAPIKey(apiKey))
// Write a Document (Upsert)
result, err := client.Collection("todos").Documents().Upsert(ctx, typesenseTodo)
All the above has to be triggered by a user action. When the user creates a new Todo like in the UI image below, the Controller will create a new model and save it to the database.
Again, note that I'm using Eloquent to perform the save.
public function store(request $request): redirectresponse
{
$todo = new todo;
$todo->id = Uuid::uuid4()->toString();
$todo->name = $request->name;
$todo->description = $request->description;
$todo->save();
return redirect('/todos')->with('status', 'todo data has been inserted');
}
Searching the Collection
There are some great resources for syncing Typesense when using Scout. However, what I love about building things is that there are usually multiple ways of solving a problem. And what I really enjoy with the DynamoDB pattern is that everything happens in its own isolated space. The write operation to DynamoDB happens, and the user is returned control from the API call. The synchronization process happens asynchronously from the user's action. By having the Stream trigger the Lambda function, that workload is also isolated.
With that said, using the Scout integration, my
Todo
model would expose search functions. That isn't
what I'm going to show below. I need to use the same Typesense API
that I used in the Lambda function, but instead of Golang, I'm
going to use good ole PHP.
Laravel Provider
To use the client in a consistent and reliable fashion, let's
create a Provider
so that I can inject it into my
TodoController
.
php artisan make:provider SearchProvider
<?php
namespace App\Providers;
use Illuminate\Support\ServiceProvider;
use Symfony\Component\HttpClient\HttplugClient;
use Typesense\Client;
class SearchProvider extends ServiceProvider
{
public function register(): void
{
$this->app->singleton(Client::class, function ($app) {
$client = new Client(
[
'api_key' => '<API Key>',
'nodes' => [
[
'host' => '<Typesense Host>',
'port' => '443',
'protocol' => 'https',
],
],
'client' => new HttplugClient(),
]
);
return $client;
});
}
}
Injecting and Searching
With a SearchProvider
established that exposes the
Typesense Client, I can now begin to handle search operations from
my Web App. I start by injecting the Client into my controller.
private Client $client;
public function __construct(Client $client)
{
$this->client = $client;
}
Once configured, I can then handle search operations in a controller method.
public function search(request $request): view
{
$search = '';
if ($request->search) {
$search = $request->search;
}
$values = $this->client->collections["todos"]->documents->search(
[
'q' => $search,
'query_by' => 'name,description',
'sort_by' => 'created_at:desc',
]
);
$searched = [];
foreach ($values['hits'] as $todo) {
$item = $todo['document'];
$t = new Todo;
$t->id = $item['id'];
$t->name = $item['name'];
$t->description = $item['description'];
$t->created_at = $item['created_at'];
$t->updated_at = $item['updated_at'];
array_push($searched, $t);
}
log::debug($searched);
return view('todo')->with(['todos' => $searched]);
}
Notice in the query_by
property of the search
criteria that I'm going across multiple fields. Doing something
like this in DynamoDB would be tough to accomplish but with
Typesense, it's extremely simple. The search parameters allow for
even further customization such as supplying the search criteria
and the sort order. What I like about using Typesense is that I'm
not having to prepend or append %
to make some kind of
wildcard syntax. Typesense handles that for me. It also handles
things like typographical errors which I'll show you below.
In addition, you get what hit, the score, and some other useful items in the result. That extra data gives you the ability to produce richer UIs and user experiences.
When searching for the word Demonstration
I
correctly see the one record that I created a few paragraphs ago.
That all makes sense. I could also have used Demo
or
stration
or any other contains
type
clause.
However, what about when a user makes a mistake. They meant to
search for Demo
, but they typed Deom
.
Wouldn't it be nice if your search provider could make that
correction? Well Typesense does!
These types of features are the reason that you choose a purpose-built provider when storing your data.
Solution Recap
And that wraps up the code and the user experience. To quickly recap what we've built:
- A PHP Laravel Application
- Create Todos
- Search Todos
Thoughts and Impressions
First off, I started building for the web in the late 90s when PHP was competing with Perl's CGI. I'm a huge fan of the community and ecosystem, and it continues to get better and better each year. Laravel adds something magical to the full application experience. I love that I can mix and match components and attach just the things I want based upon my user requirements.
Which is precisely why including Typesense into a project is both so simple and so powerful. Laravel gives me the power through Eloquent to save my data where I want, and then I can make decisions about what to do from there.
My thoughts on using Laravel with DynamoDB and Typesense can be summarized like this:
- Saving models with Eloquent is seamless
- Using AWS DynamoDB Streams fits into my desire to build Cloud-Native solutions that scale and meet the highest level of service demands
- Using the PHP Typesense client as a Provider was a breeze and injecting it via Dependency Injection into the Controller is exactly how I'd expect it to work.
- The Typesense API is clean, easy to use, and powerful. The fact that I get multiple field searching, hits, ranks, and sort is just amazing. And add in that I can run the server as a managed component in their cloud is awesome.
The one thing that I wished the Typesense cloud offered was a purely serverless and consumption-based model. With DynamoDB and Lambda, I only pay for the resources I consume and that's either per data written/read or time spent computing. As opposed to the Typesense cloud where I pay for capacity upfront, which might leave unused capacity that I'm wasting during my billing period. It's on me as the builder to manage that and effectively using the resources correctly.
With all that said, that isn't a reason I wouldn't use the Typesense cloud. It works amazingly well and much easier than running the service myself.
Wrapping Up
Thanks so much for reading about an alternative approach to saving and synchronizing a Laravel-based web application with a Typesense search datastore. I've run through a good bit of code and configuration of which I can't possibly demonstrate in an article.
So here are the repositories that I worked from. Feel free to clone and run for yourself. If you've got any feedback, please leave them in the repositories.
Building software for customers is my life's work. I enjoy seeing delighted users that don't have to worry or think about the complexity that is lurking just under the surface. Choosing Laravel, AWS, and Typesense are great foundational decisions from which to start a build and accomplish those abstractions.
They come with rock solid platforms, libraries, and outstanding communities. It's never been a better time to be a part of the PHP and Laravel community!
Thanks again for reading and happy building!
The post Serverless Search using Laravel and the Typesense Cloud appeared first on Laravel News.
Join the Laravel Newsletter to get all the latest Laravel articles like this directly in your inbox.
Read more https://laravel-news.com/serverless-search-laravel-typesense