Increased Latency on LangSmith Run Ingestion, Frontend, and API

Resolved·Degraded performance

From approximately 1716 UTC to 1751 UTC the LangSmith UI and Run Ingestion experienced degraded performance.

During this time, all runs submitted to LangSmith were successfully ingested but may have been delayed in appearing in the LangSmith UI.

We identified a set of inefficient queries related to filtering in the Threads view for Projects that, when run against very large projects, were starving our production database of resources. We have shipped optimized versions of those queries that significantly improve performance and reduce CPU and memory utilization and have not seen a recurrence.

Wed, Aug 7, 2024, 02:59 AM

(11 months ago)

·

Affected components

Aug 6, 2024, 05:31 PM

06:25 PM

LangSmith Application

LangSmith API

LangSmith Run Ingestion

Updates

Resolved

From approximately 1716 UTC to 1751 UTC the LangSmith UI and Run Ingestion experienced degraded performance.

During this time, all runs submitted to LangSmith were successfully ingested but may have been delayed in appearing in the LangSmith UI.

We identified a set of inefficient queries related to filtering in the Threads view for Projects that, when run against very large projects, were starving our production database of resources. We have shipped optimized versions of those queries that significantly improve performance and reduce CPU and memory utilization and have not seen a recurrence.

Wed, Aug 7, 2024, 02:59 AM

Monitoring

Latency has returned to historic norms. We are monitoring and continuing to investigate the root cause of the slowdown.

Tue, Aug 6, 2024, 06:25 PM(8 hours earlier)

Investigating

We have received reports of increased latency in traces appearing in LangSmith and slow loading or errors in the LangSmith Frontend and are actively investigating the issue.

Tue, Aug 6, 2024, 05:31 PM(54 minutes earlier)