In the previous part „How Did We Fight Microsoft Office 365 Support (Part 1/2): The Nightmare Begins “ I’ve re-told our story with Microsoft Office 365 EOP. Like the entire Office 365, it’s a great product, when it works. If there is a problem, a “free” English lessons with people from all around the world await you (trying to find something positive about it). 🙂
Trying to go around
It has been a week since we’ve had problems with EOP, and we have no support at all. So we turned to our partner “Tech Data”, through which we buy CSP (as an indirect CSP partner we have a connector between us and MS). In fact, it is in their interest that everything works. They promised to try to escalate over “the acquaintance” directly at Microsoft. We gave them all of the details and waited for help.
As it happens, the answer was that they can not help us. It is most likely that our ISP blocks it. The positive thing is, they do not think it’s our fault. While official support, ignoring the evidence, continues to blame both router and Exchange customers.
We have called the ISP, who said was not blocking anything. We did not seem to get anywhere on this road either. It occurred to us, however, that the customer still has a back-up connection via another ISP and maybe we could try the EOP there. We asked Tech Data whether they will be able to do something if we try it through another ISP and the results would be the same. They confirmed that it would no longer be a problem.
We got everything set up and set up an EOP via backup connection. Because we had a problem when validating the connector, it was clear that the problem would persist. We chose randomly delayed emails again, collecting logs (interestingly, we got faster and faster with each repetition). We have also done tracert from both connections to prove that they do not share the path together (see figure – the route meets at the MS Office 365 network).
With newly loaded ammunition, we turned back to Tech Data. Unfortunately, for a few days, they were silent only to tell us that they could not do anything.
Back at the front line
By trying to go around with Tech Data, we could not move any further. So we have continued our unequal fight with Microsoft Office 365 support and provide them with test information through another ISP.
- Our arguments:
- The error is not on our side: we have a packet capture from the ISP router, which shows that at the time the e-mail from O365 to Exchange has failed, there were no attempts to connect the O365.
- The error is not on the ISP side: we tried to set EOP over to another ISP that does not share any points till the Microsoft Network with the first one.
I would expect that such arguments cannot loose. Microsoft has asked us to shut down the TLS (communications encryption) server and send them a complete packet capture (including payload). If nothing is found, they would open an “internal request” (I do not know what it is, but it sounds like a solution to our problems). 🙂 With the vision of change in the communication, we have pulled our sleeves upwards, got everything and sent it to MS.
It’s gone! Two weeks after we started solving the problem, we forced Microsoft to start looking for a bug within itself.
Never celebrate early
Days have gone, and we have only received daily reports that there is no new ticket information. It took six days since the “internal request” was opened before a change has occurred.
Now, hold on… they wrote (after six days!) If we could not update the firewall rules, add some new IP addresses and try again. Apparently, they probably have a problem with passing the information, because packet capture from ISP has eliminated any problem on the firewall (when the connection does not even get to our router, it does not matter what I have set up in there). At that time, we did not use EOP at the same time – we could not wait for repair again. So we have SMTP enabled without limitation. We´ve tried it again nonetheless and gave them all the information.
If you think that a bigger “joke” from Microsoft’s internal team (I imagine there is a lot of high-end professionals in there) couldn´t come, so you’re mistaken. An additional 6 days have passed and the answer came. We will freely share excerpt from our communication, because I think you will really enjoy it. 🙂
- Microsoft: „Product Engineering Team has finished the investigation. In the backend SMTP logs we can see that connections from the Finland Datacenter can be established, while connections from the Austrian datacenter are refused.“. Basically, after 2 weeks they found out that we have problems with EOP (since we tried two ISPs and each has another route to the MS, the error will probably not be on our side).
- Me: „Nice to know that. However, what we are supposed to do with this information? As I have written:
- we don’t have any IP filter in place
- we tried it over two different ISP with the same result (tracert was included before)
- probably the problem is somewhere in Microsoft network (msn.net)“.
- Microsoft: „This is the response from our internal escalation team, basically there is nothing that we can do from our side. “
- Me: „This doesn’t sound like solution for me. The problem is still present, and we need to fix it. You found only what is wrong, but as I have told you, the problem cannot be on our side and on side of our ISP.“
- Microsoft: „The internal team has detected that the connection to your servers cannot be established. We cannot do troubleshooting on your local environment. “
- Me: „Previously I have provided you with a capture of SMTP traffic from our ISP. It proves that the failed connection attempts didn’t at all reach our router. We have also tried it through different ISP, who doesn’t share any network path with the first ISP, but the problem was still the same. Because of this I think the problem is not on our side. “
- Microsoft: „Once again, the answer of the internal team is final. As Andrei has already specified, we don’t have direct contact with them and this is the answer that we get. According to the logs they retrieved the connection that is sent from the cloud, but your server is rejecting the connection. Again, the answer is final and there is nothing that I or Andrei can do further on this. “
And as he writes, it was really over. Problem closed. Maybe I have sent a complaint to their superiors (I can not remember it and I couldn´t find it in the emails).
Force Majeure
You might think the story ended badly. After two days, however, they wrote to us from the support, if we still want to resolve it, so let’s try it again. In truth, I do not even know why we have tried it again, but we did it … and it all happened at once. What wizards are at the support. They did not do anything, we did not do anything, and it suddenly works … some kind of miracle.
Finally, we received another email from the support: „Since we’ve managed to find a proper resolution for the issue faced, I’ll go ahead and archive this case. “– and I thought they did not change anything. „At Microsoft Online Services we continually strive to deliver the highest levels of support and your feedback is extremely valuable in helping us identify where we can improve further. “– thank you, guys. The best user experience!
Unfortunately, like the story of the Vodafone, we do not know where the error was. I think the connection was rejected by some of their routers through which the data has flowed or their forwarding servers were overloaded. But we were glad that the problem was gone and we can concentrate on other projects.
Conclusion
As I have already written in another article, I do not like these problems at all. I am quite skeptical about the support lines and I imagine martyrium every single time, where there is a presumption of error on your side. It is also unpleasant that no one pays the time lost. We can not request the customer to pay to repair the technology we have recommended and it did not even work. And wanting money for Microsoft … it would be nice, but we most likely will not be able to. We had to pay for all of it. Normally, these costs would be covered by the commission of sale (but this is so small for the CSP program that it will be repaid for as a mortgage – in a few decades).
What experience do you have with Microsoft support? Could you share some “tricks” on how to escalate the problem or come up with a faster solution?