This incident has been resolved.

An issue with token counting on a limited number of runs caused a resource consumption loop that caused increased latency in run ingest starting at approximately 1445 UTC and caused all authenticated API calls to fail with 403 errors between 1545 UTC and 1625 UTC.

Traces submitted prior to approximately 1545 UTC will have been successfully ingested. Traces submitted between 1545 and 1625 UTC were rejected by LangSmith with a 403 Unauthorized error.

A hotfix was put into place that addressed the issue at approximately 1625 UTC which has mitigated the original issue.

Run ingest, API and frontend access have been functioning normally since that time.We are continuing to monitor for a recurrence and are working to put steps in place to prevent a recurrence.

We have an ongoing issue that is impacting the ingest of runs and also causing intermittent issues with the LangSmth Front End. We are actively working on a fix.