Hi!
We have a very strange problem from a customer site this morning. There was an issue with TFS on one workstation so we asked the customer to upgrade to our latest version (packaging also the TF 4.0.7.1). All client and server side DLL are properly upgraded now.
But now the workstations cant connect to the server and get a lease. It only works from the server alone.
The log file is set to show all notifications and has a lease time of 30 seconds for debugging purposes.
We see that one faulty workstation is asking for all the leases at once! It looks like something is wrong on the client side when receiving the lease. It thinks there is an error and ask again. This is my supposition. Then it throws an error that there is no more leases available.
Originally they had 3 licences. We increased it to 5 to see what is going on. Here is the result.
2017-01-03, 09:20:36 <notification>: New connection from IP: xxxx::xxxx:xxxx:a53:b521%112017-01-03, 09:20:36 <notification>: New lease assigned (PCNAME, 1). Expires: 2017-01-03 14:21:06 (in UTC). Used / Total leases: 2 / 52017-01-03, 09:20:36 <notification>: New connection from IP: xxxx::xxxx:xxxx:a53:b521%112017-01-03, 09:20:36 <notification>: New lease assigned (PCNAME, 1). Expires: 2017-01-03 14:21:06 (in UTC). Used / Total leases: 3 / 52017-01-03, 09:20:36 <notification>: New connection from IP: xxxx::xxxx:xxxx:a53:b521%112017-01-03, 09:20:36 <notification>: New lease assigned (PCNAME, 1). Expires: 2017-01-03 14:21:06 (in UTC). Used / Total leases: 4 / 52017-01-03, 09:20:36 <notification>: New connection from IP: xxxx::xxxx:xxxx:a53:b521%112017-01-03, 09:20:36 <notification>: New lease assigned (PCNAME, 1). Expires: 2017-01-03 14:21:06 (in UTC). Used / Total leases: 5 / 52017-01-03, 09:20:36 <notification>: New connection from IP: xxxx::xxxx:xxxx:a53:b521%11
Within a single second, all remaining leases have been consumed by the workstation!
What is going on? We have no clue. I did not see any firewall of virus software installed on the workstation, except Microsoft stuff packaged in Windows 10 (Defender, etc.). Our software works well if run directly on the server, but it does not work on two workstations. Firewall rules are set properly (by TFS) on the server.
Can it have anything to do with IPv6? Just before the upgrade of TFS the log file was filled with IPv4 addresses.
Thank you very much for any help,Alexandre Leclerc
Hey Alexandre,
That's some very odd behavior. We're going to look into it. Any chance we can get access to the workstation?
Also, what client version is the customer using (what version of the TF library)?
>> "Can it have anything to do with IPv6? Just before the upgrade of TFS the log file was filled with IPv4 addresses."
Unlikely. TF 4.x is the first version to fully support IPv6.
Hey Wyatt,
I got information from the customer that after lunch all started to work, for no apparent reasons. They did nothing else and we did nothing else.
The last thing we did at the customer site (and it was still not working) was to restart the TFS service to use the new config file (notifications and lease time at 60 seconds).
After lunch we removed the IPv6 binding from the local network adapter of the server. (After it started to work again, but we did it anyway.)
So there is unfortunately no more problematic PC to play with. Its sad because I would have liked to find the source and reason of the problem to avoid it in other customer sites.
I leave it to your investigation if you want, but I have nothing to let you work on for debugging. Sorry.
Thank you for your answer and best regards!Alexandre
We're going to keep digging into this, but was there any chance the customer was on a VM?
Hi Wyatt,
The chances would be extremely low. They use stock PC upgraded to Windows 10. The server is in fact a standard PC too.
Thanks Wyatt and best regards,Alexandre
Edit 1: Fixed a typo.
Hi Wyatt,
The same ghost is back today at this client site. After a Windows Update, this specific laptop was not about to connect to TF.
2017-01-23, 14:58:44 <notification>: New connection from IP: ::ffff:10.17.65.1032017-01-23, 14:58:44 <notification>: New lease assigned (username, 1). Expires: 2017-01-23 19:59:44 (in UTC). Used / Total leases: 2 / 32017-01-23, 14:58:44 <notification>: New connection from IP: ::ffff:10.17.65.1032017-01-23, 14:58:44 <notification>: New lease assigned (username, 1). Expires: 2017-01-23 19:59:44 (in UTC). Used / Total leases: 3 / 32017-01-23, 14:58:44 <notification>: New connection from IP: ::ffff:10.17.65.103...2017-01-23, 14:59:44 <notification>: Lease has expired (Joany, 1). Used / Total leases: 2 / 32017-01-23, 14:59:44 <notification>: Lease has expired (Joany, 1). Used / Total leases: 1 / 3
All available licences are issued at once to that laptop. If there were 10 free leases, it would consume all 10. In this log file there was already one user connected. So TF issued the last 2 available to the same laptop that was trying to connect.
I have no clue on what is going on. Could it be a hardware problem with the Laptop Wifi?
Thank you for your kind help and have a nice day,Alexandre
Hey Alexandre,
In an early 4.0.x build of TurboFloat we introduced a regression in the VM detection where all machines were marked as a VM even if they weren't. Normally this would be a non-issue, and for 99.99% of use-cases it is a non-issue, but it seems this one customer of yours is requesting many license leases simultaneously.
They might be accidentally launching multiple instances of your app at the same time and thus requesting multiple leases concurrently before the TurboFloat Library has a chance to "share" information with the other instances of your app.
At any rate, we've fixed this bug in 4.0.9.5, out now.
Thanks for reporting this. This was a tricky one to track down.
Hi Wyatt,
Thank you very much for your follow-up on this. I am confident that they do not launch many instances (I tested it myself). But who knows what is happening! I will plan a TF/TA upgrade this week to include the latest changes and fixes in our package. Hopefully this will help this very specific customer, with this very specific laptop.
Thank you very much and best regards,Alexandre Leclerc
