Re: Better Database Visibility
Marc-Andre Lemburg <[email protected]> Mon, 16 Sep 2024 21:01:04 +0200
| Newsgroups | gmane.comp.python.db |
|---|---|
| Organization | eGenix.com Software GmbH; http://www.egenix.com/ |
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============0935429623898536498== Content-Type: multipart/alternative; boundary="------------E0Qw0088B0GasSa7L7Wqm3dI" Content-Language: en-US, de-DE This is a multi-part message in MIME format. --------------E0Qw0088B0GasSa7L7Wqm3dI Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 16.09.2024 20:12, Jean-Mark Wright wrote: > Hi Marc-Andre, > > Thanks for your response. > > My request isn't directly related to Python. I reached out because > we're mostly a Python shop. Python is the primary space I'd be looking > to solve this problem for. I figure some folks would understand some > of the intricacies and challenges of achieving my goals and could help > me with pointers. Possibly I could make some amendments to the > database interface, or something else. We're using Postgres. > > > There are plenty tools out there for monitoring and tracing databases > > out there, but those are typically not Python-specific. > > We're using DataDog and switching providers isn't probable currently. > There's some space to augment the information we currently have. I'm > not aware of all the possibilities though. That's why I've reached out. PostgreSQL provides plenty of statistics to monitor and insight tooling: https://www.postgresql.org/docs/current/monitoring-stats.html https://www.postgresql.org/docs/current/monitoring.html You can access these using psycopg from Python, of course, and write custom logic to alert you. Or you can use Grafana or a similar tool to visualize them (via Prometheus or OpenTelemetry): https://grafana.com/solutions/postgresql/monitor/ It's also possible to ingest the data in OpenTelemetry, Nagios/Icinga, Zabbix, Checkmk, etc. There also plenty of commercial tools or commercial offerings for the OSS tools out there. Most of these (if not all), give you alerting, visuals, and data recording. This is a vast space and where to dig deeper depends a lot on your use cases, architecture and data sizes. E.g. if you're into tracing query plans, you could have a look at PG Ferret: https://github.com/ChrisBellew/pg-ferret Now, getting all this data is easy. What is hard, is making sense of it :-) > > On Sat, Sep 14, 2024 at 5:00 PM Marc-Andre Lemburg <[email protected]> wrote: > > Hi Jean-Mark, > > this list is about Python database connectivity. How does your > request > related to Python ? You also don't mention the database in question. > > There are plenty tools out there for monitoring and tracing databases > out there, but those are typically not Python specific. > > Thanks. > > > On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote: > > Hi, > > I'm aiming to improve database trace quality to better > understand what's > > happening in our database. I'd love to get index information, query > > planning information, performance statistics and richer errors > into our > > database spans. I don't know what the best layer is for this > enrichment > > (e.g. database module, OpenTelemetry, in our application, etc...). > > > > I'm looking for the following specifically: > > > > * Thoughts on the possibilities of getting this information > and where > > it would come from? > > * Thoughts on whether this enrichment aligns with the database > > specification's purpose? > > * Any contacts who could help answer or provide more questions. > > > > > > This article > > > <https://jaywhy13.hashnode.dev/dear-editor-we-need-better-database-observability> > captures the issues I've faced in greater detail. > > > > Thanks in advance for your help! > > > > > > -- > > *JEAN-MARK **WRIGHT* | Staff Engineer > > > <https://s3.amazonaws.com/wave-buoyant/public/Wave_logo_RGB-109x30.svg> > > > <https://www.waveapps.com/><https://www.waveapps.com/><https://www.waveapps.com/><https://www.waveapps.com/><https://www.waveapps.com/> > > > > Join our community onFacebook > <http://www.facebook.com/waveHQ>,LinkedIn > > <http://www.linkedin.com/company/1196866>, orInstagram > > <https://www.instagram.com/wave_hq> > > > > This message and any attachments are intended only for the use > of the > > addressee and should be considered confidential. If you are not an > > intended recipient, you may not review, copy or distribute this > message. > > If you have received this communication in error or would like > to stop > > receiving these emails, please notify the sender by replying to > this > > email. Wave is located at 500-155 Queens Quay East, Box 3, > Toronto ON, > > M5A 0W4. > > > > _______________________________________________ > > DB-SIG maillist - [email protected] > > https://mail.python.org/mailman/listinfo/db-sig > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Sep 14 > 2024) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and > costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > > > -- > *JEAN-MARK **WRIGHT* | Staff Engineer > <https://s3.amazonaws.com/wave-buoyant/public/Wave_logo_RGB-109x30.svg> > <https://www.waveapps.com/> > > Join our community onFacebook > <http://www.facebook.com/waveHQ>,LinkedIn > <http://www.linkedin.com/company/1196866>, orInstagram > <https://www.instagram.com/wave_hq> > > This message and any attachments are intended only for the use of the > addressee and should be considered confidential. If you are not an > intended recipient, you may not review, copy or distribute this > message. If you have received this communication in error or would > like to stop receiving these emails, please notify the sender by > replying to this email. Wave is located at 500-155 Queens Quay East, > Box 3, Toronto ON, M5A 0W4. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 16 2024) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ --------------E0Qw0088B0GasSa7L7Wqm3dI Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">On 16.09.2024 20:12, Jean-Mark Wright wrote:<br> </div> <blockquote type="cite" cite="mid:CAA4YZkQsbp68ucDbDWd_8bOYViya2k6tbzg+XRm=CpmdF_DanA@mail.gmail.com"> <div dir="ltr"> <div>Hi Marc-Andre,</div> <div><br> </div> <div>Thanks for your response.<br> </div> <div><br> </div> <div>My request isn't directly related to Python. I reached out because we're mostly a Python shop. Python is the primary space I'd be looking to solve this problem for. I figure some folks would understand some of the intricacies and challenges of achieving my goals and could help me with pointers. Possibly I could make some amendments to the database interface, or something else. We're using Postgres.</div> <div><br> </div> <div>> There are plenty tools out there for monitoring and tracing databases <br> > out there, but those are typically not Python-specific.</div> <div><br> </div> <div>We're using DataDog and switching providers isn't probable currently. There's some space to augment the information we currently have. I'm not aware of all the possibilities though. That's why I've reached out.<br> </div> </div> </blockquote> <p>PostgreSQL provides plenty of statistics to monitor and insight tooling:<br> </p> <p> <a class="moz-txt-link-freetext" href="https://www.postgresql.org/docs/current/monitoring-stats.html">https://www.postgresql.org/docs/current/monitoring-stats.html</a></p> <p> <a class="moz-txt-link-freetext" href="https://www.postgresql.org/docs/current/monitoring.html">https://www.postgresql.org/docs/current/monitoring.html</a><br> </p> <p>You can access these using psycopg from Python, of course, and write custom logic to alert you.</p> <p>Or you can use Grafana or a similar tool to visualize them (via Prometheus or OpenTelemetry):</p> <p> <a class="moz-txt-link-freetext" href="https://grafana.com/solutions/postgresql/monitor/">https://grafana.com/solutions/postgresql/monitor/</a></p> <p>It's also possible to ingest the data in OpenTelemetry, Nagios/Icinga, Zabbix, Checkmk, etc.</p> <p>There also plenty of commercial tools or commercial offerings for the OSS tools out there.<br> </p> <p>Most of these (if not all), give you alerting, visuals, and data recording.</p> <p>This is a vast space and where to dig deeper depends a lot on your use cases, architecture and data sizes.</p> <p>E.g. if you're into tracing query plans, you could have a look at PG Ferret:</p> <p> <a class="moz-txt-link-freetext" href="https://github.com/ChrisBellew/pg-ferret">https://github.com/ChrisBellew/pg-ferret</a></p> <p>Now, getting all this data is easy. What is hard, is making sense of it :-)<br> </p> <p><br> </p> <blockquote type="cite" cite="mid:CAA4YZkQsbp68ucDbDWd_8bOYViya2k6tbzg+XRm=CpmdF_DanA@mail.gmail.com"><br> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Sat, Sep 14, 2024 at 5:00 PM Marc-Andre Lemburg <<a href="mailto:[email protected]" moz-do-not-send="true" class="moz-txt-link-freetext">[email protected]</a>> wrote:<br> </div> <blockquote class="gmail_quote">Hi Jean-Mark,<br> <br> this list is about Python database connectivity. How does your request <br> related to Python ? You also don't mention the database in question.<br> <br> There are plenty tools out there for monitoring and tracing databases <br> out there, but those are typically not Python specific.<br> <br> Thanks.<br> <br> <br> On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote:<br> > Hi,<br> > I'm aiming to improve database trace quality to better understand what's <br> > happening in our database. I'd love to get index information, query <br> > planning information, performance statistics and richer errors into our <br> > database spans. I don't know what the best layer is for this enrichment <br> > (e.g. database module, OpenTelemetry, in our application, etc...).<br> > <br> > I'm looking for the following specifically:<br> > <br> > * Thoughts on the possibilities of getting this information and where<br> > it would come from?<br> > * Thoughts on whether this enrichment aligns with the database<br> > specification's purpose?<br> > * Any contacts who could help answer or provide more questions.<br> > <br> > <br> > This article <br> > <<a href="https://jaywhy13.hashnode.dev/dear-editor-we-need-better-database-observability" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://jaywhy13.hashnode.dev/dear-editor-we-need-better-database-observability</a>> captures the issues I've faced in greater detail.<br> > <br> > Thanks in advance for your help!<br> > <br> > <br> > -- <br> > *JEAN-MARK **WRIGHT* | Staff Engineer<br> > <<a href="https://s3.amazonaws.com/wave-buoyant/public/Wave_logo_RGB-109x30.svg" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://s3.amazonaws.com/wave-buoyant/public/Wave_logo_RGB-109x30.svg</a>><br> > <<a href="https://www.waveapps.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.waveapps.com/</a>><<a href="https://www.waveapps.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.waveapps.com/</a>><<a href="https://www.waveapps.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.waveapps.com/</a>><<a href="https://www.waveapps.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.waveapps.com/</a>><<a href="https://www.waveapps.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.waveapps.com/</a>><br> > <br> > Join our community onFacebook <<a href="http://www.facebook.com/waveHQ" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">http://www.facebook.com/waveHQ</a>>,LinkedIn <br> > <<a href="http://www.linkedin.com/company/1196866" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">http://www.linkedin.com/company/1196866</a>>, orInstagram <br> > <<a href="https://www.instagram.com/wave_hq" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.instagram.com/wave_hq</a>><br> > <br> > This message and any attachments are intended only for the use of the <br> > addressee and should be considered confidential. If you are not an <br> > intended recipient, you may not review, copy or distribute this message. <br> > If you have received this communication in error or would like to stop <br> > receiving these emails, please notify the sender by replying to this <br> > email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, <br> > M5A 0W4.<br> > <br> > _______________________________________________<br> > DB-SIG maillist - <a href="mailto:[email protected]" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">[email protected]</a><br> > <a href="https://mail.python.org/mailman/listinfo/db-sig" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://mail.python.org/mailman/listinfo/db-sig</a><br> <br> -- <br> Marc-Andre Lemburg<br> eGenix.com<br> <br> Professional Python Services directly from the Experts (#1, Sep 14 2024)<br> >>> Python Projects, Coaching and Support ... <a href="https://www.egenix.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.egenix.com/</a><br> >>> Python Product Development ... <a href="https://consulting.egenix.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://consulting.egenix.com/</a><br> ________________________________________________________________________<br> <br> ::: We implement business ideas - efficiently in both time and costs :::<br> <br> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48<br> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg<br> Registered at Amtsgericht Duesseldorf: HRB 46611<br> <a href="https://www.egenix.com/company/contact/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.egenix.com/company/contact/</a><br> <a href="https://www.malemburg.com/" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://www.malemburg.com/</a><br> <br> </blockquote> </div> <br> <br> <span class="gmail_signature_prefix">-- </span><br> <div dir="ltr" class="gmail_signature"> <div> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div><span><b>JEAN-MARK </b></span><span><b>WRIGHT</b> </span>| <span>Staff Engineer</span></div> <div><a href="https://s3.amazonaws.com/wave-buoyant/public/Wave_logo_RGB-109x30.svg" target="_blank" moz-do-not-send="true"><img moz-do-not-send="true"></a></div> <div><a href="https://www.waveapps.com/" target="_blank" moz-do-not-send="true"><img moz-do-not-send="true"></a><br> <br> </div> <div>Join our community on<span> </span><a href="http://www.facebook.com/waveHQ" target="_blank" moz-do-not-send="true">Facebook</a>,<span> </span><a href="http://www.linkedin.com/company/1196866" target="_blank" moz-do-not-send="true">LinkedIn</a>, or<span> </span><a href="https://www.instagram.com/wave_hq" target="_blank" moz-do-not-send="true">Instagram</a></div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div><span><br> </span></div> <div><span>This message and any attachments are intended only for the use of the addressee and should be considered confidential. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error or would like to stop receiving these emails, please notify the sender by replying to this email. Wave is located at 500-</span><span>155 Queens Quay East, Box 3, Toronto ON, M5A 0W4.</span></div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> <pre class="moz-signature" cols="72">-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 16 2024) >>> Python Projects, Coaching and Support ... <a class="moz-txt-link-freetext" href="https://www.egenix.com/">https://www.egenix.com/</a> >>> Python Product Development ... <a class="moz-txt-link-freetext" href="https://consulting.egenix.com/">https://consulting.egenix.com/</a> ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 <a class="moz-txt-link-freetext" href="https://www.egenix.com/company/contact/">https://www.egenix.com/company/contact/</a> <a class="moz-txt-link-freetext" href="https://www.malemburg.com/">https://www.malemburg.com/</a> </pre> </body> </html> --------------E0Qw0088B0GasSa7L7Wqm3dI-- --===============0935429623898536498== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ DB-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/db-sig --===============0935429623898536498==--