I have another update on the camera shipments.
First a bit of a background in the issue.
As I have explained in earlier posts, we have been dealing with an issue from our sensor supplier where the glass that covers the front of the sensor has been contaminated and they have been working on that issue. They realized they had a contamination issue that turned out to be caused by the packaging of the glass was shipped to their factory and so that contaminated glass was used on the sensors and sent to us.
Now there were a few fundamental problems with this. Firstly we should never have received parts that had contaminated glass. It turned out their quality control software at the sensor company was poorly designed. Secondly it had taken them months to work out what was going on and to get new packaging to ship the glass to the factory mounting it on the sensors. It’s been months.
However a few weeks ago things were looking good and the supplier got glass that was clean, they updated their test software to correctly test the sensors and could start shipping sensors to us again.
Now as I mentioned last week, the problem is while we did start getting sensors that passed our quality control when used in building our cameras, a lot of sensors did not pass. This was confusing because the sensor’s supplier was supposed to have fixed their test software and had new clean glass.
Working out what was going wrong is what we have been busy doing over the last week. It’s going to get a little technical here, however I think everyone wants to know what’s going on, instead of platitudes.
In our frustration over still getting sensors that would not pass our quality check, we decided to move in and completely audit the sensor supplier’s process in detail using our engineers. We wanted to fully understand what was going on.
We wanted to know how the supplier could the fix to get clean glass not be working 100% and how could the sensor supplier still let bad parts through their quality control process and ship them to us, in the belief that they were “good” parts? This was very confusing. I mean, fixing the contaminated glass should have been a quick simple job so how could they still not get it right after months of work?
What we found when investigating their processes was quite surprising. Of course we had known the original problem with their quality control checks was their test software had not been modified for color sensors. In the past their sensors were used for scientific use and used in black and white. Also their glass was never used as other customers bonded the lens optics onto the sensor itself. In our case we use the sensor in a conventional way and the customers change lenses. We need the glass on the sensor like all other cameras do.
Also, they had never built a camera using the sensor they make for us. We are the only camera that’s used this sensor and glass combination. It’s like designing and building cars but no one at the company has every driven one.
So it turns out their quality process is really only good at testing the semiconductor die. It’s no good at testing the quality of the overall sensor product with the glass in front. This meant they could not even see the problems we were seeing, so that’s why we were getting bad parts. We sent them the information on how to build our test setup and yesterday they started testing using it. Now they are seeing the same quality problems we are seeing. This is good as it means we should not get any more bad sensors.
The problem left is that out of a test batch of 30 sensors, only 4 worked well enough so we can build cameras using them. This is bad. So while the good news is they can now see the same problems we see, the question is why is there still contamination on the glass.
The reason is the contaminated glass issue in many ways distracted them from the problems their manufacturer is having bonding the glass to the sensor itself. The sensor supplier now has two sources of glass, and both of them are showing the same problems. The parts without glass are ok, and the problems appear when the glass is bonded to the sensor. If the glass is clean then it’s really the company bonding on the glass that are introducing contamination.
Now the amazing part is that the first batch of sensors we got that we used for developing the camera and that were fine when we started production were manufactured by a completely different company to the second and subsequent batches of sensors. I could not believe this news when I heard it today as it explains a lot.
Our current understanding is that the company that has been bonding on the glass is crap and they have been contaminating the glass when bonding it. Because the sensor suppliers test process was also bad, it meant that no one really knew what was going on and it’s been weeks and weeks of confusion.
The sensor supplier is getting some new sensors made at the original supplier, which we should get test data back on late this week. Once we see this we will know if the original supplier can make the parts without contamination and so we can start building cameras again. I don’t know why they changed glass bonding companies.
I am sorry this is a really long way of explaining what’s going on. It’s a complex issue and the only way to explain what’s going on is to actually explain what’s going on in detail. It’s been hard to update at times because there has been so much confusion at times about these issues and if they have been fixed or not. We don’t know until we build a bunch of cameras.
What has really shocked me is how long it has taken our sensor supplier to fix this. They have been very bad at moving quickly and really thinking about what’s going wrong. If we had not moved in with our engineers, they still would have no idea what was going on. It’s taken months and driven us crazy with frustration.
So the current plan is to get some sensors from the original glass bonding company and based on their upgraded testing we should know more at the end of the week if we are going to get a good supply of sensors starting to ship using that new company.
I will let you all know later this week or early next when we get some of these sensors to build cameras with and will know if we can start production full speed again.
Lastly, please take it easy on our PR folks. They want more regular updates and it’s me personally that are stopping that, because I don’t want to do fluff updates that don’t say anything and I don’t want to lie to people.
Of course PR people want to do regular updates, but each stage in solving this problem has taken our supplier more than a week or so of work before we hear anything new, and then often we get more questions, not answers. It’s been frustrating, but our PR guys are only trying to help. There is some really crazy stuff being said, but at this point can only provide the info to you as we get it.
Sorry for the long update but I am just brain dumping the situation as it is today so you know what’s going on. I hope it helps.