Simplifying Laravel Log Querying with JSON Formatting and DuckDB
Logging is a critical aspect of application development; without effective logging, diagnosing issues can be like working in the dark. Laravel's default log formatting, whilst human-readable, can be challenging to query without complex regex patterns.
[2024-05-08 21:06:18] local.DEBUG: Debug log line {"hello":"world"}
Imagine hovering over the fire of a customer support ticket, needing to quickly determine why a purchase hasn't been processed correctly. In such situations, you need to be able to rapidly rule in or out the web application as the culprit. Computers excel at rapid tasks, but only if they can easily process the data you provide them.
Enter JSON formatting. Laravel can format your log lines in JSON using the JSON formatter. Here's how you can configure it in your config/logging.php file:
'channels' => [
...
'single' => [
'driver' => 'single',
'path' => storage_path('logs/laravel.log'),
'level' => env('LOG_LEVEL', 'debug'),
'formatter' => Monolog\Formatter\JsonFormatter::class,
],
]
With JSON-formatted logs, you open up a world of possibilities for querying and analysis. Whilst there are numerous software options available for parsing JSON, I highly recommend installing DuckDB. As a Laravel Developer, you likely already possess SQL skills. DuckDB allows you to query JSON logs using a syntax very similar to SQL.
Here's an example:
> duckdb -c "SELECT message, context, level_name, channel, strptime(datetime, '%xT%X.%n+00:00') as datetime FROM read_json_auto('storage/logs/laravel-2024-05-08.log')"
┌────────────────┬───────────────────────┬────────────┬─────────┬────────────────────────────┐
│ message │ context │ level_name │ channel │ datetime │
│ varchar │ struct(hello varchar) │ varchar │ varchar │ timestamp │
├────────────────┼───────────────────────┼────────────┼─────────┼────────────────────────────┤
│ Debug log line │ {'hello': world} │ DEBUG │ local │ 2024-05-08 21:06:18.000178 │
└────────────────┴───────────────────────┴────────────┴─────────┴────────────────────────────┘
This query will return a table with columns for message, context, level_name, channel, and datetime, making it easy to analyse and filter your log data.
What's more, you can export the output to a CSV file and share it with all stakeholders.
By adopting JSON formatting for your Laravel logs and leveraging the power of DuckDB, you can quickly and easily access your log data when you need it most. This approach will save you valuable time and effort during those critical moments when rapid diagnosis is essential.