[{"data":1,"prerenderedAt":424},["ShallowReactive",2],{"/2025/10/30-amazon-aurora-serverless-the-provider-did-not-return-a-providermanifesttoken-string-error":3},{"id":4,"title":5,"body":6,"date":415,"description":103,"extension":416,"meta":417,"navigation":418,"path":419,"robots":420,"seo":421,"stem":422,"__hash__":423},"posts/2025/10/30 Amazon-Aurora-Serverless-The-provider-did-not-return-a-ProviderManifestToken-string-error.md","30 Amazon Aurora Serverless The Provider Did Not Return A ProviderManifestToken String Error",{"type":7,"value":8,"toc":412},"minimark",[9,18,25,28,31,34,36,39,41,44,46,49,51,54,67,69,72,74,77,80,82,85,87,90,104,106,109,116,119,127,130,138,141,149,152,161,164,173,176,185,188,190,193,199,207,209,215,223,233,235,248,261,263,266,275,277,280,288,290,304,312,315,324,326,332,340,342,345,347,350,352,355,364,372,375,377,380,383,386,389,398,401,409],[10,11,13],"post-title",{":date":12},"date",[14,15,17],"h1",{"id":16},"amazon-aurora-serverless-the-provider-did-not-return-a-providermanifesttoken-string-error","Amazon Aurora Serverless \"The provider did not return a ProviderManifestToken string\" Error",[19,20,21],"notes",{},[22,23,24],"p",{},"This is a repost from my old blog. First posted in 8/26/2019.",[26,27],"br",{},[22,29,30],{},"We had a good working application which connect to Amazon Aurora Serverless pretty well and just recently it started to intermittently unable to connect with the error \"The provider did not return a ProviderManifestToken string\". The following is some spec of the application:",[22,32,33],{},".NET Framework 4.6.2\nMySQL.Data 6.10.9\nMySQL.Data.Entity 6.10.9\nEntityFramework 6.2.0 (EF6)",[26,35],{},[22,37,38],{},"When I debugged the application, it has inner exception which has the message: \"Sequence contains more than one matching element\". Upon more troubleshooting, I remember that Aurora Serverless is a cluster and it requires at least 2 subnets which reside in 2 different Available Zones. Amazon does not recommend the use of IP address instead provide us with an endpoint to connect to the cluster. That means, the endpoint might resolve to two IP addresses.",[26,40],{},[22,42,43],{},"So, I decided to see what DNS lookup will show and indeed, the endpoint resolves to two IP addresses. Hence, my suspect is the MySQLConnection was expecting a single IP and receive two instead from the URL and thus throws an error. I tried to replace the endpoint in the connection string with one of the IP addresses and no error, but that is not the expected solution.",[26,45],{},[22,47,48],{},"Additionally, there are intermittent errors with the following message:\nAuthentication to host '...' for user '...' using method 'mysql_native_password' failed with message: Reading from the stream has failed",[26,50],{},[22,52,53],{},"With more reading, I stumbled upon the following article:",[22,55,56],{},[57,58,61],"span",{"className":59},[60],"text-blue-600",[62,63,64],"a",{"href":64,"rel":65},"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html",[66],"nofollow",[26,68],{},[22,70,71],{},"At the end of the articles, it says \"If you use the mysql client to connect, currently you must use the MySQL 8.0-compatible mysql command.\"",[26,73],{},[22,75,76],{},"I then realized that I have been using MySQL.Data version 6.10.9 and that might need an update. However, there is no MySQL.Data.Entity with major version of 8 which I found out later that it has been changed to MySQL.Data.EntityFramework for .NET Framework application (for .NET Core application, use MySQL.Data.EntityFrameworkCore). So I put the endpoint back into the connection string and update the Nuget packages to:",[22,78,79],{},"MySQL.Data 8.0.20 8.0.17\nMySQL.Data.EntityFramework 8.0.20 8.0.17",[26,81],{},[22,83,84],{},"And this time, the application is finally able to connect successfully. As a side note, MySQL Workbench with major version of 8 that can connect just fine.",[26,86],{},[22,88,89],{},"Additionally, we need to have a custom DbExecutionStrategy to handle the error that might happen, possibly because of cut-off connection due to scaling, and retry. In my case, it is like the following:",[91,92,93],"code-block",{},[94,95,100],"pre",{"className":96,"code":98,"language":99},[97],"language-text","Public Class MyCustomDbExecutionStrategy\n        Inherits DbExecutionStrategy\n\n    Sub New(maxRetryCount As Integer, maxDelay As TimeSpan)\n        MyBase.New(maxRetryCount, maxDelay)\n    End Sub\n\n    Protected Overrides Function ShouldRetryOn(exception As Exception) As Boolean\n        Return exception IsNot Nothing AndAlso exception.Message.Contains(\"Reading from the stream has failed\")\")\n    End Function\nEnd Class\n\nAnd the custom DbExecutionStrategy has to be applied manually as follow:\n\nDim executionStrategy As New MyCustomDbExecutionStrategy(5, TimeSpan.FromSeconds(15))\nexecutionStrategy.Execute(\n    Sub()\n        Using dbContext = New MyDbContext()\n        ...\n        End Using\n    End Sub)\n","text",[101,102,98],"code",{"__ignoreMap":103},"",[26,105],{},[22,107,108],{},"Please refer to update July 2, 2020 and July 6, 2020 for additional information.",[22,110,111,115],{},[112,113,114],"strong",{},"Updates","\nNovember 14, 2019\nFor somewhat reason, updating the Nuget packages to version 8.0.18 brings back the error, so we stay with 8.0.17 for now.",[22,117,118],{},"November 27, 2019\nI have a bit of time to play so I checked if the new version really doesn't work. After upgrading it from 8.0.17 to 8.0.18, it truly doesn't work. The following are the error messages and stack traces 3 layer deep exception:",[91,120,121],{},[94,122,125],{"className":123,"code":124,"language":99},[97],"Top level exception\n\"The provider did not return a ProviderManifestToken string.\"\n\n   at System.Data.Entity.Core.Common.DbProviderServices.GetProviderManifestToken(DbConnection connection)\n   at MySql.Data.EntityFramework.MySqlManifestTokenResolver.ResolveManifestToken(DbConnection connection)\n   at System.Data.Entity.Utilities.DbConnectionExtensions.GetProviderInfo(DbConnection connection, DbProviderManifest& providerManifest)\n   at System.Data.Entity.DbModelBuilder.Build(DbConnection providerConnection)\n   at System.Data.Entity.Internal.LazyInternalContext.CreateModel(LazyInternalContext internalContext)\n   at System.Data.Entity.Internal.RetryLazy`2.GetValue(TInput input)\n   at System.Data.Entity.Internal.LazyInternalContext.InitializeContext()\n   at System.Data.Entity.Internal.InternalContext.Initialize()\n   at System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType)\n   at System.Data.Entity.Internal.Linq.InternalSet`1.Initialize()\n   at System.Data.Entity.Internal.Linq.InternalSet`1.AsNoTracking()\n   at System.Data.Entity.Infrastructure.DbQuery`1.AsNoTracking()\n   at ...\n",[101,126,124],{"__ignoreMap":103},[22,128,129],{},"First inner exception",[91,131,132],{},[94,133,136],{"className":134,"code":135,"language":99},[97],"\"Unable to connect to any of the specified MySQL hosts.\"\n\n   at MySql.Data.MySqlClient.NativeDriver.Open()\n   at MySql.Data.MySqlClient.Driver.Open()\n   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)\n   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()\n   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()\n   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()\n   at MySql.Data.MySqlClient.MySqlPool.GetConnection()\n   at MySql.Data.MySqlClient.MySqlConnection.Open()\n   at MySql.Data.MySqlClient.MySqlProviderServices.GetDbProviderManifestToken(DbConnection connection)\n   at System.Data.Entity.Core.Common.DbProviderServices.GetProviderManifestToken(DbConnection connection)\n",[101,137,135],{"__ignoreMap":103},[22,139,140],{},"Last inner exception",[91,142,143],{},[94,144,147],{"className":145,"code":146,"language":99},[97],"\"Sequence contains more than one matching element\"\n\n   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)\n   at MySql.Data.Common.StreamCreator.GetTcpStream(MySqlConnectionStringBuilder settings)\n   at MySql.Data.Common.StreamCreator.GetStream(MySqlConnectionStringBuilder settings)\n   at MySql.Data.MySqlClient.NativeDriver.Open()\n",[101,148,146],{"__ignoreMap":103},[22,150,151],{},"Searching online, I found the following thread:",[22,153,154],{},[57,155,157],{"className":156},[60],[62,158,159],{"href":159,"rel":160},"https://forums.mysql.com/read.php?38,678859,678859#msg-678859",[66],[22,162,163],{},"which leads to the following bug report:",[22,165,166],{},[57,167,169],{"className":168},[60],[62,170,171],{"href":171,"rel":172},"https://bugs.mysql.com/bug.php?id=97448",[66],[22,174,175],{},"And eventually the actual code change:",[22,177,178],{},[57,179,181],{"className":180},[60],[62,182,183],{"href":183,"rel":184},"https://github.com/mysql/mysql-connector-net/commit/9bc44843fda0c2e4aed1e22cc00c1221d17dc00b#diff-7440e953b4e85502ea58c60b17f249ee",[66],[22,186,187],{},"For now, we still stick with 8.0.17 until further fix.",[26,189],{},[22,191,192],{},"June 25, 2020\nActually from the last update until now, we still experience intermittent issues with the error. So, I did few more testing. In my case, I took the following steps to get it completely error free. Probably some steps are not necessary but I haven't had time to test what is actually necessary.",[22,194,195,198],{},[112,196,197],{},"Step 1"," (update 7/2/2020, the function somehow is not called)\nCreate custom DbExecutionStrategy. Mine is actually very simple. All I did is retry only when the error message contains \"ProviderManifestToken\". I found out MySqlExecutionStrategy that comes with the Nuget package doesn't work for my case.",[91,200,201],{},[94,202,205],{"className":203,"code":204,"language":99},[97],"Protected Overrides Function ShouldRetryOn(exception As Exception) As Boolean\n    Return exception IsNot Nothing AndAlso exception.Message.Contains(\"ProviderManifestToken\")\nEnd Function\n",[101,206,204],{"__ignoreMap":103},[26,208],{},[22,210,211,214],{},[112,212,213],{},"Step 2","\nCreate a custom DbConfiguration. This is where we can set our custom DbExecutionStrategy. My custom class derives from MySqlEFConfiguration to get all the preset goodness.",[91,216,217],{},[94,218,221],{"className":219,"code":220,"language":99},[97],"Public Class MyCustomDbConfiguration\n        Inherits MySqlEFConfiguration\n\n    Sub New()\n        MyBase.New()\n        SetExecutionStrategy(MySqlProviderInvariantName.ProviderName, Function() New MyCustomDbExecutionStrategy())\n    End Sub\n\nEnd Class\n",[101,222,220],{"__ignoreMap":103},[22,224,225,226],{},"Couple of things to watch out for:\nWe can only use one DbConfiguration per provider per app domain. I got an error when trying to assign different configuration to different DbContext within a single application.\nSomehow SetExecutionStrategy caused an error when we execute raw sql with the following message: \"Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index\". So we still use MySqlEFConfiguration for the application that has raw SQL.\nSetExecutionStrategy is marked to cause an issue with Transactions in MySql GitHub.  For more info: ",[57,227,229],{"className":228},[60],[62,230,231],{"href":231,"rel":232},"https://github.com/mysql/mysql-connector-net/blob/68c54371821c87ff40a773acc127ce357b46a5ae/Source/MySql.Data.EntityFramework6/MySqlEFConfiguration.cs",[66],[26,234],{},[22,236,237,240,241],{},[112,238,239],{},"Step 3","\nThere are 3 different ways to apply a DbConfiguration according to the MySQL Connector documentation: ",[57,242,244],{"className":243},[60],[62,245,246],{"href":246,"rel":247},"https://dev.mysql.com/doc/connector-net/en/connector-net-entityframework60.html",[66],[249,250,251,255,258],"ul",{},[252,253,254],"li",{},"Adding the DbConfigurationTypeAttribute on the context class",[252,256,257],{},"Calling DbConfiguration.SetConfiguration(new MySqlEFConfiguration()) at the application start up",[252,259,260],{},"Set the DbConfiguration type in the configuration file.",[26,262],{},[22,264,265],{},"What I didn't know is which one takes precedence over which. I found out later that the one in configuration file will take precedence over code-based:",[22,267,268],{},[57,269,271],{"className":270},[60],[62,272,273],{"href":273,"rel":274},"https://docs.microsoft.com/en-us/ef/ef6/fundamentals/configuring/code-based",[66],[26,276],{},[22,278,279],{},"In my case, I need the DbContext to default to use MySqlEFConfiguration and overwrite as necessary. So for those applications that need to use the custom DbConfiguration, I added the codeConfigurationType attribute in the config file (app.config/web.config):",[91,281,282],{},[94,283,286],{"className":284,"code":285,"language":99},[97],"\u003CentityFramework codeConfigurationType=\"namespace.MyCustomDbConfiguration, assembly\">\n",[101,287,285],{"__ignoreMap":103},[26,289],{},[22,291,292,295,296,303],{},[112,293,294],{},"Step 4","\nOn my applications that has the error, the default connection factory was set to LocalDbConnectionFactory. So, following MySQL Connector documentation, I set it to the following under the ",[101,297,302],{"className":298},[299,300,301],"bg-gray-200","p-2","rounded","\u003CentityFramework>"," tag:",[91,305,306],{},[94,307,310],{"className":308,"code":309,"language":99},[97],"\u003CdefaultConnectionFactory type=\"System.Data.Entity.Infrastructure.SqlConnectionFactory, EntityFramework\"/>\n",[101,311,309],{"__ignoreMap":103},[22,313,314],{},"I found out later that this step might not be necessary as it will only be used when there is no connection string, but I leave it that way since it doesn't hurt.",[22,316,317],{},[57,318,320],{"className":319},[60],[62,321,322],{"href":322,"rel":323},"https://docs.microsoft.com/en-us/ef/ef6/fundamentals/configuring/config-file#code-first-default-connection-factory",[66],[26,325],{},[22,327,328,331],{},[112,329,330],{},"Step 5","\nThe server that hosted my problematic applications might not have MySql connector installed, so according to the MySQL Connector documentation, I should have added the following entry in the config file:",[91,333,334],{},[94,335,338],{"className":336,"code":337,"language":99},[97],"\u003Csystem.data>\n   \u003CDbProviderFactories>\n     \u003Cremove invariant=\"MySql.Data.MySqlClient\" />\n     \u003Cadd name=\"MySQL Data Provider\" invariant=\"MySql.Data.MySqlClient\" description=\".Net Framework Data Provider for MySQL\" \n          type=\"MySql.Data.MySqlClient.MySqlClientFactory, MySql.Data, Version=8.0.17.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d\" />\n   \u003C/DbProviderFactories>\n\u003C/system.data>\n",[101,339,337],{"__ignoreMap":103},[26,341],{},[22,343,344],{},"Watch out for the version number. It needs to match the MySql.Data.dll assembly version.",[26,346],{},[22,348,349],{},"So far, the error has not re-appeared in my case. Also, there is a new version 8.0.20 Nuget packages as of this update. I haven't tested it extensively so I can't vouch for its compatibility for now. However, I might write an update once I am certain it doesn't cause issue in our applications.",[26,351],{},[22,353,354],{},"July 2, 2020\nThe steps on update June 25, 2020 was not working somehow. When we have a lot of processes hitting the database, the error message returns. I did a quick check and found out that the ShouldRetryOn function was not called. I ended up manually calling the execution strategy.",[22,356,357],{},[57,358,360],{"className":359},[60],[62,361,362],{"href":362,"rel":363},"https://docs.microsoft.com/en-us/ef/ef6/fundamentals/connection-resiliency/retry-logic#solution-manually-call-execution-strategy",[66],[91,365,366],{},[94,367,370],{"className":368,"code":369,"language":99},[97],"Dim executionStrategy As New MyCustomDbExecutionStrategy()\nexecutionStrategy.Execute(\n    Sub()\n        Using dbContext = New MyDbContext()\n        ...\n        End Using\n    End Sub)\n",[101,371,369],{"__ignoreMap":103},[22,373,374],{},"On the Nuget version, I found out MySQL.Data and MySQL.Data.EntityFramework version 8.0.20 can connect to Aurora Serverless just fine.",[26,376],{},[22,378,379],{},"July 6, 2020\nAfter diving deeper into the code, I had to make a few adjustment and finally managed to ensure that the exception was properly handled.",[22,381,382],{},"First, I reanalyzed the exception details and found out the inner exception is slightly different than the one specified in update November 27, 2019 with the following message:",[22,384,385],{},"Authentication to host '...' for user '...' using method 'mysql_native_password' failed with message: Reading from the stream has failed",[22,387,388],{},"Second, by the time the exception reaches ShouldRetryOn method, it has been unwrapped.",[22,390,391],{},[57,392,394],{"className":393},[60],[62,395,396],{"href":396,"rel":397},"https://github.com/dotnet/ef6/blob/master/src/EntityFramework/Infrastructure/DbExecutionStrategy.cs",[66],[22,399,400],{},"Third, I have to update my ShouldRetryOn to check for the error message above instead of checking for \"ProviderManifestToken\" string since it is the inner exception will be passed to ShouldRetryOn. In my case, I prefer to check for \"Reading from the stream has failed\" string with hope to capture broader errors.",[91,402,403],{},[94,404,407],{"className":405,"code":406,"language":99},[97],"Protected Overrides Function ShouldRetryOn(exception As Exception) As Boolean\n    Return exception IsNot Nothing AndAlso exception.Message.Contains(\"Reading from the stream has failed\")\")\nEnd Function\n",[101,408,406],{"__ignoreMap":103},[22,410,411],{},"Looking into the log file of my application, it finally retries fine and the application ran without issue.",{"title":103,"searchDepth":413,"depth":413,"links":414},2,[],"2025-10-30T00:00:00.000Z","md",{},true,"/2025/10/30-amazon-aurora-serverless-the-provider-did-not-return-a-providermanifesttoken-string-error",null,{"title":5,"description":103},"2025/10/30 Amazon-Aurora-Serverless-The-provider-did-not-return-a-ProviderManifestToken-string-error","b4jUupdNzQQthv5MzjpFDKy60rXQ6OgrK2nFzQih7Zk",1785167452722]