In keeping with the vendor-theme, this next story is about a company that we will call Fortune-Teller (FT). FT has software in every device you own, whether you use Apple iOS/OSX, Android Phones, Windows PCs, Linux, etc. They are everywhere, and the guy who started the company looks like he could be the villain in the next Die Hard movie. I am not sure if there is a moral to this story, or if I am just sharing it so people can start to see how “these things go”, as they say. In either case, lets get started.
FT, along with every other conceivable product, offers an Enterprise Service Bus solution. In a complex architecture, many different systems talk to each other and sometimes things get over-complicated and pieces start to break as systems undergo change. An ESB essentially acts as a middle-man between these services. Services call the bus instead of calling each other directly, so each service just needs to know how to talk to the bus instead of 4-5 other services. The bus generally can handle a bunch of other stuff like security or error-handling as well. The company I worked for at the time had decided that they wanted FT’s solution, and like any other vendor product we bring in, I decided to take a peek.
FT’s ESB solution is not as simple as some of our other vendor products. It is broken into many different web applications and web services, all running on VMs in a managed network. There are several administrator consoles, and we wanted to take a look at each one. These consoles have various purposes, like shutting off and starting web services, deploying web services, incident and event management, and configuring the authentication. Being that FT is a fairly large company with a CSO and a dedicated security team, we didn’t expect to find much.
We started testing, and not long in we found CSRF in two key areas: the security-focused console and the web-service configuration console. The first console allowed us to shut off access to services, create administrator users, and open up services to anonymous access. The second console allowed us to turn off logging and monitoring on services and upload our own services (essentially creating a backdoor into the server).
Of course, the first thing we did after discovering and confirming the issues was contact the vendor. Unfortunately, in this case we did not have vendor-people on site so we were emailing back and forth with a remote team. This has its pros and cons. On one side, the vendor had people across the globe ready to respond at any time of day. On the other side, we had to repeat the same explainations multiple times to different people as we were shuffled across the different teams at FT. Of course, FT was no exception in the bug-grief process: first comes denial and then comes a full working demo of the exploit (note: this time we used kittens instead of puppies).
I would say that FT was generally more receptive, but the convenience of around-the-globe support slowed things down as each support-centre wanted to confirm the issue for themselves each time we were shuffled to the next team. Of course, this was augmented by the fact that each support team also had multiple tiers that needed to confirm before their development teams also wanted to confirm the issue. On the bright side, you get fairly good at explaining an issue after that many passes at it.
After everyone and their dog agreed on the issue and agreed on a fix, FT passed the issue on to the team responsible for actually fixing it and our visibility went black. After a week we contacted the support team for an update, who had to pass the message on through the chain, only to reply (several days later) with something to the effect of “it will come when it comes, it is planned for a future release”.
Anyways, take all that and times it by two because we found essentially the same issue in two separate consoles, and each console is a separate team.
One day, we will find a vendor who accepts the words of experts and delivers timely fixes. Today is not that day.