Looking for the EU status page? Find it here: https://eu.status.smith.langchain.com
Looking for the EU status page? Find it here: https://eu.status.smith.langchain.com
Resolved
This incident has been resolved.
Monitoring
An issue with token counting on a limited number of runs caused a resource consumption loop that caused increased latency in run ingest starting at approximately 1445 UTC and caused all authenticated API calls to fail with 403 errors between 1545 UTC and 1625 UTC.
Traces submitted prior to approximately 1545 UTC will have been successfully ingested. Traces submitted between 1545 and 1625 UTC were rejected by LangSmith with a 403 Unauthorized error.
A hotfix was put into place that addressed the issue at approximately 1625 UTC which has mitigated the original issue.
Run ingest, API and frontend access have been functioning normally since that time.We are continuing to monitor for a recurrence and are working to put steps in place to prevent a recurrence.
Investigating
An infrastructure issue is blocking authentication for LangSmith and resulting in 403 errors on all services. We are actively working to address the issue.