Remote Sinks in pgwatch3 by Akshat Jaimini

As a mentor for Google Summer of Code (GSoC) 2024, I am thrilled to witness the innovative ideas and technical expertise that young minds bring to the table. This year, I have the pleasure of working with Akshat Jaimini on an exciting project that aims to enhance PostgreSQL monitoring through the development of RPC sinks for pgwatch3. Akshat's work focuses on extending pgwatch3 by developing a mechanism to allow Remote Procedure Call (RPC) sinks, which will offer more flexibility and integration possibilities for users.

At this point, I would like to hand over to Akshat, who will delve into the specifics of the project, its goals, and the challenges we aim to address. Akshat, the floor is yours!

Hello! I am Akshat Jaimini, a Computer Engineering Student from India. I was a part of PostgreSQL for the Google Summer of Code 2023 Program where I worked closely with the pgweb development team on developing a testing harness suite for the Official PostgreSQL website. Excited to be a part of the PostgreSQL community!

Sinks?

pgwatch v3 is well under works and with that comes the possibility of new and awesome features that were not there earlier. One such feature is the addition of ‘Remote Sinks’ in this awesome tool.
Sinks…?

Well if you have used pgwatch2 earlier, you would be familiar with the storage concept. Let me get you up to speed.

Previously we configured pgwatch in two ways: the Push and the Pull configs. First one will send store measurements somewhere directly, while the pull config allows scrapers to fetch measurements from pgwatch.

Regardless of the config being used, essentially all your metric measurements need to be stored somewhere. That particular ‘storage unit’ is referred to as a Sink in pgwatch v3.

So basically pgwatch fetches metric measurements from the ‘Source’ Database and sends/stores them in ‘Sinks’.

Why Remote Sinks?

Till now pgwatch supported PostgreSQL, JSON file and Prometheus as the possible sink solutions. Although these are great for the primary use case but they fall short when these metrics are required for something more.

Although these new formats can be added as the feature requests come in, but why keep the capability to ourselves? So we introduced the concept of Remote Sinks which basically provides you with an interface using the go RPC library to get the metrics measurements as a stream of data and utilize them for your respective use cases. This project is a part of Google Summer Of Code, 2024.

So the users get an option to write any type of sink without worrying about if pgwatch will be able to integrate with it.

Architecture

As I mentioned earlier, we decided to go with the default RPC library that comes in Go. To visualize this we can refer to the following diagrams:

There are some key takeaways from the architecture above:

• Remote sinks can be called via a single Remote Procedure Call, therefore no additional processing required.
• pgwatch does not care about the details of the remote sink at the back.
• No additional changes required in pgwatch3 setup.

To use your remote sinks you can simply run:

You can try out the development build on GitHub.

As of now the demo supports the following output formats: CSV and Parquet. And we plan on adding more examples for our users but that's a surprise we’ll like to keep for pgconf.eu 2024 😉

How can I develop my own sinks?

To develop your own sinks you can follow the given steps.

Clone the repository from https://github.com/destrex271/pgwatch3_rpc_server

In the sinks folder create a new file with whatever sink format name you want and declare an enum for your sink in the sink/types.go  file. Also don’t forget to update the conditionals! (We’ll end up changing this so that it automatically prepares calls according to the enums in the future)

With this your setup is complete!

PS: You can raise a PR if you feel that these sinks can be used by other people too!

While developing your sinks you can use the primary_receiver  instance passed into the UpdateMeasurement()  function of your sink for the following:

• Retrieving Meta parameters associated with the Remote Procedure Call.
• Receiving Sync Metric Signals(Like ‘DELETE monitored database’, ‘ADD new metric’, etc…) if you want to support them in your sinks.

How do I test my sink?

Worry not, I have a dummy client written within the repository which you can use to generate requests for your newly developed sink. It is there to simulate pgwatch3 without actually launching an instance.
Road Ahead!

The development on this part of pgwatch3 is in full swing and we hope to soon make it as easily accessible for all our beloved users!

Also if you wish to contribute let’s meet in the discussions/issues section of the project.

Thanks for reading!

In order to receive regular updates on important changes in PostgreSQL, subscribe to our newsletter, or follow us on Facebook or LinkedIn.