Posts tagged “manufacturing”
You are currently viewing a selection of posts from the Pololu Blog. You can also view all the posts.
Nearly two years of operations under the COVID-19 pandemic are behind us. Like many other businesses around the world, our biggest challenges have moved from direct health and safety concerns to secondary disruptions, most notably the supply chain issues and the global chip shortage that has been making news and shutting down factories since last year. Initially, we were relatively isolated from the shortages because we had maintained a high inventory, often stocking a year or more of critical components. However, as the disruptions dragged on, our reserves were depleted, and we have had to resort to increasingly drastic measures to keep operating at all.
I apologize to our customers who are frustrated by our worsening response times, price increases, and unavailability of products. I hope showing you some of what we are dealing with will make it easier to understand.
Here is a screenshot from our internal system showing the inventory history of a relatively unremarkable component (a small MOSFET) that we have been using for almost ten years now:
Inventory history for a component with shortages in 2021.
The stock history is representative for a typical component that we gradually put into more product designs so that the rate of usage keeps increasing and the amount of stock we keep on hand gradually increases, too. Usage of this part ramped up in 2018, to around 35 thousand pieces per year. We last received some shipments in mid-2020 that put us in a seemingly-secure place, but the situation became less comfortable as we got into 2021, and the past several months have been downright alarming since we might only have enough parts for two more months, while the estimated shipment dates for my orders are well into 2022. And this is with us putting the brakes on parts usage!
Slowing down component consumption is really not fun since our main options are just not making any of a product at all (sometimes we are forced into that option anyway once we run out of parts) or raising prices. Higher prices can make it confusing for customers to select among alternatives since we expect the more expensive product to generally be the better one. To help communicate that some products’ prices and availability are temporarily distorted, we added several rationing-related entries to our list of product status designations. You can see the status of each product along with stock and pricing information:
We initially focused on reducing volume discounts, and building the rationing designations into our system let us automatically exclude rationed items from sales and other special promotions. It has been almost six months since we started officially designating products as rationed, and unfortunately, what we expected to be a temporary measure for a few select items has gradually affected more and more products as component shipments keep getting delayed.
Here is the inventory history for another component, a more expensive and specialized part than the previous one:
10-year inventory history for a component being rationed in 2021.
There were already some supply issues with the part in 2017 that led us to keep slightly higher inventory of that component, and you can see the change in the pattern as we ramped up our rationing efforts. We buy the part on reels of 1,000 pieces, so the upward jumps in the graph are multiples of a thousand, and we used to use up fairly high quantities in each manufacturing run, so the downward jumps were fairly sizable, too. For instance, we might make five hundred of a product at a time and just put it all in stock on the website and not make more for a couple of months. Starting in the second quarter of this year, you can see how the inventory graph is a lot smoother as we made much smaller production runs to preserve flexibility in which products to use the components.
This strategy does help us maximize the usage of the parts we have on hand, but it comes with many costs. Production is less efficient since the machine setup is the same whether we are making twenty units of a product or a thousand units. We also have much more internal scrutinizing, planning, and checking of which products to make since accidentally making the wrong product is a much bigger problem than it used to be—it could now mean prolonged inability to make a different, otherwise unrelated product. It’s also more difficult on customers who do want to buy in bigger volumes since we used to have more stock available on our website, and customers could just order. Now, when we show 29 of a motor controller in stock and a customer needs 50, they have to talk to us about how soon we can make the additional 21. This also strains our support staff resources and reduces the service quality for all customers. And the sad thing is that we are doing a lot more work to produce and sell fewer products.
What inventory do we do have?
You might be surprised to hear that our total inventory is actually at an all-time high. And apparently, that is fairly common, even among the biggest companies, including the main electronics distributors. When I was talking to my Arrow Electronics rep last month, he said his warehouse is full. I asked of what, and he said he didn’t know, but apparently not the parts he needs.
I spend a lot of time trying to understand what we do have. We have thousands of unique components, and on average we have thousands of each one, so we have many millions of components to keep track of. Most products use many different components, and most components get used in many different products. If we are missing one part out of fifty to make a product, we can’t make the product. And usage rates for the same component are different in different products; what are we supposed to do when we have five thousand left of a component that we use in a $5 product that used to sell thousands of units a month and in a $100 product that sells hundreds of units a month, and the earliest estimated delivery of more components is eight months out? So far, we have mostly raised the prices on the $5 product, sometimes very substantially, while not changing the price on the $100 product, and that lets us keep some finished products available to offer.
There are more and more components that have been on order for over a year now, and meanwhile estimated ship dates for new orders are well into 2023 (not 2022!). It’s a scary time to be an electronics manufacturer.
Other cost increases
As I mentioned, we are going through a lot more effort to make fewer completed products, and that contributes to increased costs and higher prices independent of what we are doing with rationing. On top of that, prices for most of the components we have been using for a long time have risen substantially, even as our order volumes increase. Most increases have been in the 10% to 20% range, but several are 50% or more.
Then there are parts that we now buy in smaller quantities from catalog distributors like Digi-Key and Mouser (when we find stock there), and those prices can be several times higher. Some parts I bought a year ago for twelve cents each in quantities of fifty thousand are now costing 25 cents each in those bigger volumes, and if I order just a few hundred or a few thousand, they can cost a dollar each. If we just need one of those on a product we sell for $100, it’s not that big of a deal, but if there are three of those components on a product that used to cost $5, the price is going to have to go up, sometimes dramatically.
Non-electronics component and material costs are also going up, though those have generally been in the more modest 5% to 20% range, but shipping costs are up a lot, so that disproportionately affects heavier and bulkier items. We have had to reprice some of our stepper motors primarily because of the shipping costs to get them here, while we have thus far been able to avoid raising costs on our micro metal gearmotors (though volume discounts are smaller than they used to be). Most of our products involve at least some processing in the US, but we are able to ship some items directly from our China warehouse to other countries to reduce the impact of shipping costs and the tariffs on many products coming into the US from China.
Broader problems and delays
The broader supply chain issues are a problem, too, even though it’s not as bad as with the chip shortages. Most of our mechanical parts, from injection molded plastic parts to motors and servos, come from China, and we are more directly involved in getting them shipped here (unlike the electronics parts, which are also mostly made overseas but which we buy from American distributors who deal with getting the parts into the US). Fortunately, most of our components are small and light so we ship a lot by air anyway, but we do ship heavier and bulkier items by boat and have had our share of days looking at all the ships waiting off the coast of California and wondering when ours would finally get to dock. It seems like regular delays by various carriers like FedEx and UPS are also getting worse, and we have now had at least a couple of instances where really important parts we were waiting months for made it to the US or even to Nevada and then got lost.
I have been writing mostly about components and how it affects electronics we manufacture, which is most of our business, but the other small manufacturers whose products we resell are in the same environment, and so we are seeing price increases and extended unavailability of products from them, too.
Delivery delays and other problems are affecting our shipments to our customers, too. Unfortunately, we are again mostly at the mercy of the large shipping companies and the general situation that has led to reduced service levels around the world. Many of the providers have suspended guarantees of delivery times or extended the times they say delivery will take. We have recently added UPS to our standard shipping offerings during checkout, so our customers at least have some more options in case one service is particularly bad in their area.
When will it get better?
As I wrote a few years ago, we buy our electronics components through major authorized distributors, and we have so far not had to resort to going to secondary sellers and brokers (with the associated risks of ending up with counterfeit parts). From my talking to manufacturers’ representatives, my impression is that the semiconductor companies are just genuinely facing a combination of increased demand and reduced capacity as the pandemic interfered with their operations that are spread throughout the world. For example, ST was telling me about one motor driver that gets the silicon processed in Italy, then tested (still in silicon wafer form) in Singapore, and then chopped up and packaged in Malaysia. In this one instance, the silicon is done, and as operations resume in Malaysia, they should be able to get me some of the parts by the end of the year. But for other parts from the same company, such as microcontrollers, they don’t even have enough allocated to my general western North America region to meaningfully talk about where in the queue we are.
When I first heard predictions in early 2021 that the chip shortages would drag on through the end of the year, I didn’t really believe it. It’s increasingly clear that those predictions were right, but at least 2022 is not that far away anymore! And while we do have many orders with expected ship dates two years out (late 2023), we also have several with expected ship dates in early 2022 (and some parts have been trickling in, so it’s not as if we were completely choked off on all supply).
As we approach the holiday season when we traditionally have our biggest sale, we are assessing which products we can make and possibly discount. We have a few new releases this year that we are very excited about, but new products are especially difficult to ramp up, especially if they use new components we didn’t already have on order a year ago.
Thank you for your continued business and support
Despite the various challenges presented by the evolving pandemic and associated disruptions, we have generally been able to keep operating relatively smoothly this year. I know there are many small businesses of all types struggling or even having to shut down completely, and I am very grateful that we have avoided such extreme scenarios. Thank you to all of the employees at Pololu for so reliably keeping everything running, and thank you to all of our customers for your continued business. I wish everyone a safe and happy conclusion to the year and look forward to things improving on all fronts in 2022.
Installation of our new Mycronic MY600 that arrived earlier this month is going smoothly. Here are a few pictures and a video from our first test print on a Pololu PCB panel.
Ninoos from Mycronic showing us how to use the MY600 solder paste jet printer.
First Pololu test panel in our new Mycronic MY600 solder paste jet printer.
The gantry is supported only on the left side!
First print on a Pololu PCB panel using Mycronic MY600 solder paste jet printer.
MY600 jet printer first print: Looks like we need a little more solder paste under the central chip and a little less on the leads.
That footage of the jet printer in action is not sped up!
I am a little proud (but mostly embarrassed) that I still do basically all of the electronics component purchasing for Pololu. Today I am writing about buying components because their prices have a huge impact on the end price of our products, especially when we work to cut down other costs as I have been discussing lately. Buying parts when trying to compete globally is more frustrating than you might think, and I hope that writing this will help trigger some conversations that will help us do better and also encourage component manufacturers and distributors to better support small electronics manufacturers in the United States.
I buy almost all of the electronic components that go into our products from suppliers in the United States. That includes integrated circuits, discrete semiconductor products like transistors and diodes, resistors, capacitors, inductors, and so on. The only parts I do not buy in America are components like connectors and electromechanical devices like switches and buzzers. This post is mostly about buying components in the US from US suppliers, but I will briefly touch on the non-US components to provide some background and comparison.
There are two main reasons for those non-US components: they are much cheaper than similar parts in the US and we can evaluate that they are good enough. That second reason is important because counterfeits and fake parts are a big problem in the electronics industry. We can look at something like a 0.1" male header or an electromagnetic buzzer and see basically what it is. Once we can be confident that a component or supplier is good enough for the level of performance and reliability we need from our products, we can look at prices to see whether it’s worth the extra hassle (and that amount of hassle keeps going down) to get the components overseas, which pretty much means China (and Taiwan, in case that distinction is meaningful). And that price difference can be huge. When I started getting connectors directly from Taiwan over ten years ago, the price difference was approaching a factor of ten, meaning that for around a thousand dollars I could buy what would cost me ten thousand domestically. Some of the price differences seem to be getting smaller, but the components still seem to be easily three to five times cheaper in China. Early on (around 2005), local salespeople would ask me what prices I was getting in China, thinking they had some better deal or connection than I knew of; nowadays, they don’t even ask or try to compete.
With all the other component types, even for the most basic parts like resistors and capacitors, we are not really qualified to evaluate them, and I would not want to be responsible for doing quality control for millions of units even if we could analyze one particular instance of a resistor or capacitor and conclude that it is good enough. So, I only buy components from reputable brands through their authorized distributors, which means I buy basically all our components through American companies (with one exception I’ll get to soon that almost doesn’t feel like an exception, anyway). Those companies also tend to be the biggest electronic parts distributors in the world.
I still buy some components from the catalog-type sources that are probably familiar even to most students and hobbyists, like Digi-Key, Mouser, and Newark. Once upon a time, these companies printed large catalogs that were the best way to find out what kinds of components even existed (especially when I was growing up in Hawaii). Digi-Key stopped printing catalogs a few years ago; Newark apparently still does. In any case, these kinds of sources now have good online resources for finding parts, they tend to have a lot of parts in stock, and they are set up for small quantities, which is why they are great resources for individuals, too. Usually, it’s very easy to buy components from these sources, and I rarely interact with anyone when I do since I usually just place the orders through the web sites at their listed prices.
However, the prices usually are not the best at those most convenient sources, especially when my minimum quantities tend to be full reels with thousands of parts rather than a few individual parts. So for most of my electronics purchasing, I go with big distributors like Arrow, Avnet, and Future Electronics. These companies have local offices (though for Las Vegas, “local” tends to mean in Phoenix, Arizona or somewhere in Southern California) that have salespeople that I can talk to that can help me get lower prices. Future Electronics, being a Canadian company, is the exception that I mentioned earlier; but, working with them is basically the same as working with Avnet or Arrow since they have the local staff and things like a distribution center near Memphis, Tennessee (here is a video about it).
Back when I was a student and before I started Pololu, I thought electronics distributors just bought components from manufacturers and then sold them with some markup on their cost. I think some of that did happen and still happens today, but it’s less common and less practical now because there are so many different, specialized components that they cannot all just be sitting in stock somewhere, waiting for the unlikely scenario that someone would come along and buy them. When I buy from distributors like Digi-Key and Mouser, I am almost always buying something they have in stock; when I buy from Arrow and Future, it’s almost never something they have in stock (or it’s something they have in stock because they had good reason to expect me to order).
But there was a much bigger misconception in my naive view than just the timing of when a distributor bought some parts relative to when the end customer ordered them. The major thing I learned in the early years of running Pololu is that component manufacturers charge the distributors different prices for the same components, based on the end customer! In some ways, that’s frustrating because it means I have to do a lot more work to get a good price. I have to convince each distributor and each manufacturer to give me a good price, and sometimes I have to do that with each component. It can also be a good thing in that if I do establish a good relationship with some manufacturer and distributor, the hassle per part goes down over time as they get to know our business and what factors are important to us. And some of them probably do give me a globally competitive price sometimes.
The difficulty, and what prompted me to write this post, is with those manufacturers that do not. (That or counterfeits, which is sometimes the explanation or excuse I get back from manufacturer’s representatives.) Well, the really specific thing that led me to write this post is the AVR Freaks thread about my last post, in which I introduced our new A-Star 328PB Micro:
AVR Freaks post about A-Star 328PB Micro announcement, the motivation for this blog post.
The poster mentions a board similar to Pololu’s with an ATmega328P for $1.70 (and wonders about the chip authenticity). I went to look at the part price on Digi-Key, and this is what I saw:
Screenshot of Digi-Key price for ATmega328P with prices highlighted, retrieved 25 February 2018.
I have highlighted the relevant fields, which are the lowest prices: $1.77 each at 100 pieces, and alternately, on a reel of 2000 pieces for $1.83. 100 pieces is kind of low for a highest price break. A long time ago, I tried out the higher volume quote request option that is suggested under the 100-piece price, and after some back-and-forth, Digi-Key gave me an official volume quote for higher quantities with the exact same price as at the 100-piece break:
Excerpt from quantity price quote from Digi-Key (from around 2010)
I tried it a few more times with some other parts, but it was basically the same result every time, so I have not tried special pricing with Digi-Key since. That was close to ten years ago, so maybe it’s different now, and maybe it would be different if I had asked for 10k or 100k pieces.
Back to the original point: you can get the whole assembled board from China, in single-piece quantities, for less than the price of just the one component, even in volume quantities. And this Atmel/Microchip example is far from atypical or anywhere near the worst case. Here is an AliExpress listing for a small development board for the STM32F103C8T6 microcontroller:
AliExpress listing for an STM32F103 development board for $1.79 including shipping, captured 25 February 2018.
In this case, Digi-Key has prices listed for higher quantities than in the earlier example, and the 2400-piece reel has the best unit price of $3.04:
Screenshot of Digi-Key price for STM32F103C8T6, retrieved 25 February 2018.
I have been working on pricing for similar STM32 parts, and even at twenty thousand pieces (not through Digi-Key), they are not getting to the prices of just one of these complete boards, with double-sided assembly and a bunch of additional components (and free shipping!—though that is some separate scam that I do not blame the electronics component manufacturers and distributors for). Something clearly doesn’t add up, right?
As I mentioned earlier, some manufacturers will say the parts in the cheap boards from China are counterfeits. I believe that is the explanation in some cases, and we have our share of frustrations with knock-offs and counterfeits of our own products. But I had one experience around five years ago that makes me very skeptical that that is anywhere near the whole story. One of our more successful early products, at least by number of units sold, was a basic carrier for Allegro’s A4988 stepper motor driver. I remember being frustrated because Allegro seemed to have some deal in place with Digi-Key that gave them the best prices (and unlike the examples above, Digi-Key did have volume breaks up to many tens of thousands on that part). Digi-Key has been getting better on prices, but still, paying “Digi-Key prices” seemed like an insult when I was buying tens of thousands of these parts at a time.
It became more of a problem when the knock-offs of our boards started appearing on eBay and other sites for basically the same prices as just our component costs. I kept trying to get my suppliers to help me with my prices until I got the parts from Asia via Future Electronics. They assured me these were genuine parts through their global partners or subsidiaries, and they could do that for me because they were not authorized distributors for Allegro in the United States. So, that alone was a good indication that these parts were out there, through reputable sources, at lower prices than I could get in the US. But the conclusion to this story gets better: someone who could do something about it at Allegro finally got word that I was getting these parts elsewhere (they noticed the sales abruptly went away, and they wanted to see my invoices from other vendors) and reduced my authorized price, through Arrow, to almost half of what it had been.
This was over the course of maybe two years, and while it did help and we lowered our prices, I believe we missed out on a lot of potential sales because it took so long to get the prices down. The sad thing is that there are probably many more missed opportunities, not even just for Pololu, because of how components get priced in the US. The manufacturers seem more willing to cut prices after demand is proven and they start losing sales rather than up front to make the sales happen in the first place. And to be clear, I am not talking about small quantities like broken reels and cut tape that require extra handling and processing. I also understand that these modern 32-bit microcontrollers like the STM32 are such amazing achievements that it seems really unappreciative to complain about one costing $2. It’s just that when the same parts are costing one dollar somewhere else, we need to figure out how to get that price if we want to be globally competitive with our products.
Since I know many of you are also interested in open-source hardware, I should mention the ramifications component pricing has for openness with our designs. One of the factors that goes into how much information we release is how good I think our component price is. It’s easier to share key components that we use if I think we can keep making our products competitively.
Back to some of the comments that led me to start writing on this topic. I hope I have shown you a different kind of behind-the-scenes view than just the machines that go into making our products. Some of that “twice as expensive as a similar board from Ali / Ebay” goes all the way down to the basic component level. I hope you can see that I am working on addressing that. I would be interested in what other small manufacturers in America (and in other countries besides China) do. Should I just start sourcing more components from China? Should I be using smaller distributors in the US? (I am skeptical that would make a big difference because of my understanding that manufacturers are basically setting the prices). Or is there something else I’m completely missing? I know some of you who read my posts work for the electronics manufacturers and distributors; can you help push for getting better component pricing in the US? For all of you who like making electronics, wouldn’t it be nice to have the option (or for your kids to have the option) to do that without having to live in China or being limited to industries like aerospace and military where components costs might not matter as much?
Most of the products we manufacture are small electronics subassemblies, which we have been packaging in these pink antistatic bags that are probably familiar to everyone who has worked with electronics:
These bags are supposed to help prevent charge buildup and therefore help prevent the sensitive components inside from getting damaged just from the bags being shuffled around. But, if you charge yourself up and then grab the bag, you can still zap the electronics inside through the bag.
This year, we began transitioning our packaging to metalized shielding bags, like this:
These bags have the charge buildup prevention of the pink bags but also have a metal layer that prevent the spark from getting from your finger to the electronics parts in the bag. Most bags I have seen have some plastic coating outside the metal layer so that you don’t have zero resistance along the surface of the bag, but I have heard of people shorting things out by powering them up while sitting on some types of metallic bags.
Did a static discharge into a USB port cause this power supply to go up in smoke?
I once got a huge static shock when I plugged a flash drive into my USB port, and right after that, a bunch of sparks and smoke came out of my computer’s power supply. I’m pretty sure that wasn’t a coincidence, but I still can’t be sure it was a case of electrostatic discharge (ESD) destroying my computer. And usually, ESD damage is much more subtle and basically impossible for us to confirm. So, changing the packaging is one of those things where it will be very difficult to tell if it’s really doing anything. But we go through a lot of effort to inspect and test each item we make so that we can be confident it worked when we made it, so this packaging change should give us still more confidence that the part is in good condition when you receive it. It will take a while before all of our products get changed over, so don’t be shocked if your order arrives with a mix of packaging types.
We recently released the A-Star 32U4 Micro, which we think is the best available AVR breakout board for its size. If you are like us, you enjoy taking factory tours, seeing how things are made on How It’s Made, and watching your Krispy Kreme doughnuts get created right before you personally eat them. Since most of you have not been able to visit us here in Las Vegas, we’ve made a video that shows how your A-Star Micro gets made!
The video shows how the A-Star Micro goes from a bare printed circuit board to an assembled and tested product. It is one of our more complex boards to make because it has components on both sides—this means two trips through the stencil printer, pick-and-place machine, reflow oven, and automated optical inspection machine. Here are some of the machines featured in the video:
- Speedprint SP700 AVI stencil printer
- Samsung SM421F pick and place
- Heller 1707 MK III reflow oven
- Nordson YESTECH BX-12 automated optical inspection
For those of you who like to be mesmerized by big machines moving thousands of tiny components quickly, we also have a video that shows the full pick-and-place sequence of a panel of forty A-Star Micros. (Note that this is not an accurate representation of the assembly time since the feeders are moved to the side to make room for the camera.)
For more videos like these, see our YouTube playlist: Pololu manufacturing: how our products are made and subscribe to our channel. By the way, you can still get a free A-Star Micro with your order over $100.