We are getting a 500 error on the web application.
And the Windows App is no longer working
See below stack trace from the system.
regards,
Steve
500. Error occurred while handling your request.
URL: /Account/AuthorizeUser
Response status: 500
Response body: <!DOCTYPE html>
<html>
<head>
<title>Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.</title>
<meta name="viewport" content="width=device-width" />
<style>
body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;}
p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
.marker {font-weight: bold; color: black;text-decoration: none;}
.version {color: gray;}
.error {margin-bottom: 10px;}
.expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
@media screen and (max-width: 639px) {
pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
}
@media screen and (max-width: 479px) {
pre { width: 280px; }
}
</style>
</head>
<body bgcolor="white">
<span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>
<h2> <i>Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.</i> </h2></span>
<font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
<b> Description: </b>An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
<br><br>
<b> Exception Details: </b>System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.<br><br>
<b>Source Error:</b> <br><br>
<table width=100% bgcolor="#ffffcc">
<tr>
<td>
<code>
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>
</td>
</tr>
</table>
<br>
<b>Stack Trace:</b> <br><br>
<table width=100% bgcolor="#ffffcc">
<tr>
<td>
<code><pre>
[InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.]
System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +1329
System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) +307
System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry) +198
System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) +422
System.Data.SqlClient.SqlConnection.Open() +88004303
System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch(TTarget target, Action`2 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed) +943
System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(DbConnection connection, DbInterceptionContext interceptionContext) +503
System.Data.Entity.SqlServer.<>c__DisplayClass1.<Execute>b__0() +19
System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Func`1 operation) +234
System.Data.Entity.Core.EntityClient.EntityConnection.Open() +318
[EntityException: The underlying provider failed on Open.]
System.Data.Entity.Core.EntityClient.EntityConnection.Open() +738
System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(Boolean shouldMonitorTransactions) +167
System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction(Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) +63
System.Data.Entity.Core.Objects.<>c__DisplayClass7.<GetResults>b__5() +203
System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Func`1 operation) +234
System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption) +354
System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0() +31
System.Data.Entity.Internal.LazyEnumerator`1.MoveNext() +39
System.Linq.Enumerable.FirstOrDefault(IEnumerable`1 source, Func`2 predicate) +121
TimetrackerOnline.JointCodebase.BusinessLayer.Extensions.UserExtensions.GetUserByExternalId(IEnumerable`1 users, UserModel userModel) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.BusinessLayer\Extensions\UserExtensions.cs:26
TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.SeamlessUserManager.ValidateUser(UserModel userModel) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\SeamlessUserCreator.cs:59
TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.AuthService.ValidateOrCreateUser(NameValueCollection headers, Boolean isCreate) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\AuthService.cs:193
TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.AuthService.Authenticate(NameValueCollection headers) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\AuthService.cs:229
TimetrackerOnline.Startup.OnAuthenticateSeamless(IOwinContext context) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Startup.cs:211
TimetrackerOnline.Startup.<Configuration>b__4_0(IOwinContext context, Func`1 next) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Startup.cs:77
SharpReverseProxy.<Invoke>d__4.MoveNext() in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Lib\SharpReverseProxy\ReverseProxyMiddleware.cs:45
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +68
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<RunApp>d__5.MoveNext() +197
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +68
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<DoFinalWork>d__2.MoveNext() +184
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +117
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +510
System.Web.<>c__DisplayClass285_0.<ExecuteStepImpl>b__0() +38
System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +155
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +131
</pre></code>
</td>
</tr>
</table>
<br>
<hr width=100% size=1 color=silver>
<b>Version Information:</b> Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.3282.0
</font>
</body>
</html>
<!--
[InvalidOperationException]: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext](TTarget target, Action`2 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
at System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(DbConnection connection, DbInterceptionContext interceptionContext)
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.<>c__DisplayClass1.<Execute>b__0()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
at System.Data.Entity.Core.EntityClient.EntityConnection.Open()
[EntityException]: The underlying provider failed on Open.
at System.Data.Entity.Core.EntityClient.EntityConnection.Open()
at System.Data.Entity.Core.Objects.ObjectContext.EnsureConnection(Boolean shouldMonitorTransactions)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
at TimetrackerOnline.JointCodebase.BusinessLayer.Extensions.UserExtensions.GetUserByExternalId(IEnumerable`1 users, UserModel userModel) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.BusinessLayer\Extensions\UserExtensions.cs:line 26
at TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.SeamlessUserManager.ValidateUser(UserModel userModel) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\SeamlessUserCreator.cs:line 59
at TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.AuthService.ValidateOrCreateUser(NameValueCollection headers, Boolean isCreate) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\AuthService.cs:line 193
at TimetrackerOnline.VSTS.BusinessLayer.Services.Implementation.AuthService.Authenticate(NameValueCollection headers) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.VSTS.BusinessLayer\Services\Implementation\AuthService.cs:line 229
at TimetrackerOnline.Startup.OnAuthenticateSeamless(IOwinContext context) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Startup.cs:line 211
at TimetrackerOnline.Startup.<Configuration>b__4_0(IOwinContext context, Func`1 next) in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Startup.cs:line 77
at SharpReverseProxy.ReverseProxyMiddleware.<Invoke>d__4.MoveNext() in C:\build-agents\BuildAgent-git-1\_work\29\s\TimetrackerOnline\TimetrackerOnline.Web\Lib\SharpReverseProxy\ReverseProxyMiddleware.cs:line 45
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.<>c__DisplayClass285_0.<ExecuteStepImpl>b__0()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
--><!--
This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using <customErrors mode="Off"/>. Consider using <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> in production environments.-->
-
Hi everyone!
Thank you for your message and our apologies for the problems you are experiencing.
We've faced an issue with our cloud version of 7pace Timetracker and its database. Our database stopped accepting new connections, which led to performance issues and to an eventual outage. We took immediate steps to restore the application state and it is now operating normally again. We are continuing to investigate the core reason behind the database connection limitation and will perform hotfixes for it, to prevent such issues in the future.
About Windows Client: this outage also might had spoil auth tokens for Windows Client, so to fix it you will need to pair it again (Time -> Apps page)
Again, our sincere apologies for the inconvenience.
Please check if everything works correctly now.Regards,
Maxim -
Hello,
The ticket we created for Meher was resolved some time ago, however we decided to share the resolution in case someone will find it useful in the future.
The solution for this issue was to install version 3.1 of ASP.NET Core. Here is the link for the installation: ASP.NET Core 3.1 Runtime
Best regards,
Vanja
Please sign in to leave a comment.
Comments
5 comments