Today’s post is spawned from a tale similar to the last post, in that it deals with the subject matter of vendors and issues found in their software.
This “The Vendor” sells a provisioning platform that allows systems that need to provision services (like Order Management, Order Entry, etc.) to communicate with engineering systems using a more standardized API. The need arises partially from the fact that most engineering systems (phone switches, DACs, CMTS systems, etc.) use uncommon or low-level methods of communication to provision services and equipment which are not easy to implement in a high-level system like a web application. The Vendor solves this by allowing other applications to send HTTP requests, which it then translates into the various messages and protocols that the downstream systems understand. There is much more to it than that, but that is the simple explanation. The Vendor has their solution in over 500 companies globally, including many major ISPs, telecom companies, and the US government.
This story takes place some time in mid-2014. The company I worked for was installing a new instance of The Vendor’s solution and wanted our team to perform some testing on it. As we started testing, we were quickly finding issues all over the place. Reflected XSS on the login page, invalided redirects allowing for phishing attempts, virtually no logging capabilities or password lockouts, and more. So, as we always do, we informed the project team and the vendor.
In my experience (and sadly so), the response from vendors when reporting a vulnerability is more often than not a negative one. It seems to be a common theme, which is really too bad. In a perfect world vendors would welcome external security testing, as it helps them offer their customers the most secure software they can. The more eyes looking for problems, the more likely the problems will be found. In this case, the vendor pushed back above and beyond the average.
The issue with the highest risk (and the first issue found) was the XSS, and so I will focus on that in this post. At first, The Vendor denied the issue. This is the common first stage in the bug-grief process. As demonstrated in my previous post, there are a number of methods that can be used to work through this change: try different verbiage, describe the issue from a different point of view, explain the business or technical impacts, or give a real-life demonstration. The Vendor wouldn’t move out of the denial stage without a fight, and so we had to build a demonstration.
Our initial demo involved inserting an image of the power rangers onto the login page. While an awesome demo in theory, the non-‘evil minded’ developers at The Vendor did not see the risk. “Who cares if you can put images on the login page?” was the reply we received, despite explaining that running code is running code regardless of it being an image or a cookie-stealing script. Back to the drawing board.
We brought this full-featured demo back to The Vendor’s developers, walked them through the login process, and showed them the stolen credentials. After playing around a bit themselves, we finally had acceptance of the issue. For most, the logical thing to do would be to fix the issue that we all agree 1) exists, and 2) is a risk. In The Vendor’s case, the ‘logical’ thing to do was to write up a level-of-effort (LoE) & quote, effectively offering to ‘sell’ the fix to us.
This is the opposite of what a vendor should do. If a customer finds a security vulnerability in your software, do not attempt to charge them for a fix. You should instead fix the issue and thank your customer for assisting you in developing a better product for all your other customers.
In the end, we found other ways to mitigate the risk, but this shouldn’t have been necessary. The moral of the story: don’t rely on a vendor to give a fix, even if they understand the issue and the risk. Sometimes you need to find a way to mitigate the risk on your own.
Note: Seeing as I intentionally withheld the vendor’s name, please do not guess it in the comments section below. Also of note: Eventually, the vendor did publish a fix, but as always it is up to the customer to stay up to date on patches. Please make sure you patch often!