Moving from Relational Data to Event Sourcing: Key Insights

Introduction: Relational databases have long been the go-to choice for data storage in software development. However, in recent years, there has been a growing interest in event sourcing as an alternative approach. Event sourcing, in theory, offers several benefits, such as the ability to time travel, recover overwritten state, and reinterpret events of the past. But is it practical and suitable for all scenarios? Let’s explore some key insights from user comments to gain a better understanding of this paradigm shift.

---

The Appeal and Challenges of Event Sourcing

Event sourcing, in theory, is an appealing concept. It provides a way to maintain audit logs, store snapshots of pre-save data, and derive state from events. It offers the potential for a single system that simplifies data management and reporting. However, in practice, there are challenges that need to be considered.

One major challenge is the amount of boilerplate code required to add new CRUD workflows or implement interventions and hotfixes. Event sourcing introduces complexity that may not be suitable for early-to-mid-stage startups that frequently need to adapt and iterate quickly. It can add unnecessary complexity to existing systems and lead to maintenance and scalability issues.

Another challenge lies in handling schema changes and migrations. Changes to event data can introduce issues that may not be discovered until much later, causing conflicts and blocking important tasks. There may be temptations to make quick fixes to old events, but these changes can have unintended consequences and make reasoning about the impacts difficult.

---

The Value of Event Driven Architecture

Event driven architecture is appropriate in scenarios where something happens out of band, rather than being a traditional request/response scenario. For example, when a code push triggers a build process, the build process becomes event driven. Understanding the distinction between request/response and event driven can help determine if event sourcing is suitable for a particular use case.

---

The Role of PostgreSQL and Queues

PostgreSQL, a robust relational database system, is often suggested for storing event data. It offers powerful features and has a large community of engineers familiar with the system. PostgreSQL can handle event data until the size reaches around 2TB, at which point considerations for online storage versus summarized data come into play. There are PostgreSQL-compatible systems available for additional scaling when necessary.

In terms of scalability, queuing systems such as Amazon Simple Queue Service (SQS) can be valuable. However, SQS has eventual consistency, which can lead to duplicate messages. An alternative suggestion is to use Amazon MQ, which offers more reliable messaging.

---

Considerations for Data Storage and Indexing

Storing event data in JSONB format in PostgreSQL is a common choice. However, it’s important to consider the limitations of indexing functionality with JSONB, especially when dealing with diverse event structures and evolving event definitions. Familiarizing oneself with the documentation on JSONB in PostgreSQL can provide a better understanding of its capabilities.

When it comes to indexing, it is crucial to revise the indexing policy and remove unused or underused indexes. Indexing can have a significant impact on large inserts, and understanding the implications of index maintenance is essential for database performance.

---

When is Event Sourcing Appropriate?

Event sourcing may not be suitable for every project. It is crucial to assess the value it brings and weigh it against the potential complexity and risks involved. Some users believe that event sourcing has become a resume-boosting buzzword, leading to its implementation without clear understanding or an actual need for it. It is important to evaluate if event sourcing is the right choice based on the specific requirements and constraints of the project.

---

The Case for Traditional Relational Data

Despite the rise in popularity of event sourcing, traditional relational databases still have their merits. They offer familiarity, simplicity, and compatibility with many existing systems. Relational databases, when properly designed and optimized, can fulfill the requirements of most projects. Event sourcing may be more appropriate in specific use cases where it provides clear benefits, such as financial data management or scenarios that require time-travel capabilities.

---

In conclusion, the decision to move from relational data to event sourcing requires careful consideration of the benefits and challenges. While event sourcing offers appealing features such as time travel and state recovery, it may introduce complexity that is not suitable for all projects. Traditional relational databases still remain a reliable choice in many scenarios. The key is to understand the specific needs of the project and choose the appropriate approach accordingly.

Remember, event sourcing is not a one-size-fits-all solution, and evaluating the trade-offs is essential for making informed decisions in software development.