Listen to the podcast here:
Printer ShootOut – Shooting Down Make Magazine’s Printer Point Ranking System
We’re going to talk Make Magazine’s Printer Shootout. It’s really interesting that we’re coming in full circle because it’s been just about two years since we started the podcast and two and a half years since we got that second annual Make Magazine Printer Shootout and we finally got a 3D printer. Obviously, this came out in November, December last year, the most recent, that’s when it comes out every year in Make Magazine. It was in 2013 that we first looked at the second annual Make Magazine 3D Printer review. MakerBot was on the cover. It was the MakerBot that we ended up buying, actually at that point after that. We ended up buying it largely on information that we read in that magazine as well as other investigation we did. This was actually before there was all the publicity about the issues they had with the Smart Extruder which again, we’ve been not shy about saying we really didn’t have as many of those problems as a lot of people did. Although the Smart Extruder+ is definitely ten times better and more reliable. Anyway, the point is we did use the magazine to judge as we thought about what printer we were going to buy. We used it as a recommendation factor.
I think, actually, that was also part of the frustration. When we first got the printer, we were really pretty frustrated that it wasn’t as easy as they made it sound in the magazine. It wasn’t as good as it sounded in the magazine. The qualities we were getting weren’t good. There was a lot of things about it that didn’t match up to how we felt after reading it and deciding that this was the printer for us and then getting it home, getting it running, getting it going. It took us really about six months to get into the place at which we were like, “I could see how this is working and I could see how good this is.”
Now, several years later, as we’re reviewing the most recent Make Magazine issue about this, you can really start to understand why there was a big disconnect between what we read about in the magazine and we were experiencing. It’s really right on this first page where they’re showing the dozen or so professionals. I don’t know if they’re professional 3D printers but they’re certainly very experienced technicians, I would say, of 3D printers who, in order to review all these printers for one magazine issue. They do it in like a day or a weekend. They have to have a lot of different people operating all these machines who are very, very skilled. I think that when somebody who’s very skilled in the 3D printer industry, is using one and then writing about it, they can’t really write it from a perspective or put themselves in the shoes of a new user, which I think are the people that really read this thing the most to learn what first printer they’re going to buy. I think people experience and look at it also, but not from that same lens.
As a novice coming in or as someone new to 3D printing looking at this, it’s really a bit misleading.
They may think about what’s coming up next, what next printer am I going to buy and then it’s maybe very useful there. But as a novice coming in or as someone new to 3D printing looking at this, it’s really a bit misleading. You look at the ratings and the other things, it’s also really, to me, a disconnect between what you’re going to print and why you want a printer and the point system as well. That’s really what we wanted to talk to you guys about today. As you’re doing this, if you’re out there thinking about buying your first printer, you really weigh it against all the other pieces of information and ask more question, because I think it just leads to more questions, what you read in Make Magazine’s point system.
I actually really dislike the point system. I think it does a disservice to the reader. Because they have this point system from one to five, five being the best score in a certain category and one being the worst. They, first of all, don’t show you any pictures. It’s like, horizontal surface quality, vertical surface quality, retraction. It’s all these technical terms. To me, if your overhang wasn’t a five and it wasn’t perfect, that means you’ve got little loops. It could be suspect, it could be hard to clean up, strength problems. You could have a whole host of issues if it wasn’t a perfect five every time. It’s not giving you a sense of what that means if it’s less than that. How good or how bad was it? The point system isn’t helpful to me without seeing pictures of what was printed.
The other thing I really take issue with is they give a rating of two a pass. I don’t know about you but in my math head, when you have a two out of five, that’s 40%. I don’t see how that’s passing. I have to say, you see their little graphic that’s there, that certainly doesn’t look all that much lower than a three, like everything’s all up there. It makes it really look like they’re all way up there at the top. I know they’re adjusting for the text that’s there but it’s really a little visually deceptive in making you think that it’s better than it is. I think that that’s not helping as well.
For anyone other than a person that has a fair amount of experience already in 3D printing, the kind of information they’re providing in this magazine, I really don’t think is very helpful because they’re breaking down the different technical elements of 3D printing, like I was saying, horizontal surface quality, bridging, retraction, support material, Z Wobble, all these different things. That really doesn’t help a person considering getting into 3D printing to know, is it going to be good for printing models of jewelry, if you’re a jewelry designer? Or it’s really going to be good for printing, let’s say, toy parts could be an example. Something with tight tolerances because I need to do replace parts. It’s not giving you a sense of what you’re printing and what that means to your printing process or why you’re choosing a printer.
We say here all the time, you want to start with the what and the why. Both of those two things go hand in hand. Why are you choosing a new printer? Why are you going into 3D printing to begin with? What are you printing? Those two things are the most important decision before you look in and analyze this. Because you’re weighting these things. I don’t think that we could do something that had any amount of surface pilling or bumps or any of those kinds of things, or left a lot of strings or overhangs because of the way we do it because we want our parts to come right off of the machine with no added work. Some of these here, if they were even at a five, are they good enough? It’s really tough to know.
The other thing that I think is not really rated well or consistently across all the printers is how easy or hard it is to set up and maintain the printer in terms of calibration and all that. It’s hard enough sometimes for a new user to just get the printer properly calibrated, set-up and working properly than if you’re trying to judge if it’s producing good surface quality here or there. It may not be the actual printer’s capability that’s the issue that you’re having at that given time. It may actually be the fact that you don’t really understand yet how to properly maintain the printer and how often do you need to calibrate it, does it calibrate itself. There are so many variables in 3D printing to get a print to work well for what the printer is capable of. You’ve got to dial that in before you then experiment with all the different geometric variables there are and can this printer do that well or not.
The other thing that is not really rated well or consistently is how easy or hard it is to set up and maintain the printer in terms of calibration.
I think the articles that are written alongside with it, although they are a little short, they are really telling, for instance, on the Raise3D N2 Plus, it says it has no bed calibration. That’s an interesting thing, it’s like it’s a fixed bed, there’s no bed calibration. What does that mean and what is that about? Now I want to know more. There are lots of little details in here that are not in the rating system or in the bullet points that they track from machine to machine. I could see somebody reading that note about bed calibration as meaning that’s a bad thing. What they’re trying to say is you don’t need to. That it’s a little more, I guess, maintenance free in that. They’re calibrating the bed through software and not through hardware. That may be a good thing.
In here, in all fairness to Make Magazines, that they’re assessing it, but it’s not the same person. Each one is a different one of their technicians or judges has each taken on a different printer to talk about. One says that it’s the veteran’s machine and another one says this is a printer for educators. You’re not getting necessarily a sense of why and why they’ve chosen this one in more than a couple of little sentences. It’s easy to criticize. Certainly, they make a big effort every year to work on it. It is hard stuff to do. Having something to look at and judge is better than nothing. But what we want to communicate with you, our regular listeners, is that there are a lot of things to consider. You definitely have to read between the lines and take some of even their judgments and criticisms with a grain of salt.
The other reason we wanted to talk about this today, this is the number one question we get at WTFFF, “What printer should I buy?” It’s a very difficult question to answer. The way we approach it and the way we review printers, and for those of you who have been listening for a long time, you’ve listened to some of those reviews and checked them out on our website, we talk about printers in terms of who it would really be a good fit for, what type of products, what types of models it would be a good fit for printing, more so than the technical capabilities. There certainly are technical capabilities to cover. We show images of everything we’ve test printed on it. You get a visual comparison by what we’re seeing and what we found. Whenever there’s something anomaly or something odd or something great, we try to put a video up, so you can actually see it happening so you’ll understand what that is as well. That’s what we build into ours. This is a lot more complicated. A print magazine is more complicated. Also, you’re essentially reviewing people who are your advertisers, which is always a catchy thing.
I had that conversation once with someone, a 3D printer reseller in Southern California. We had this discussion once about the Make Magazine, the annual issue of the reviews of printers. I was talking to them, “Why didn’t they review this printer or that printer?” because that particularly year, there was actually a really big release of a printer that was highly capable. I thought, “Well, it should be in this Printer Shootout.” The thing that this reseller said to me was, “You also have to remember, Make Magazine is going to give more attention and somewhat preferential treatment to companies that are advertising in their magazine.” It was interesting because the next year after we had that discussion, there was no printer from certain manufacturers. I think there wasn’t a MakerBot even though there had been great improvements. Even this year, there was no Replicator+ which came out in the fall. I’m sure if they’d ask MakerBot for one, they would have given them one. For some reason, I guess, maybe Maker is not advertising and so it’s not in there.
I think there is a reality to that. They can only do so many printers in this weekend, even with the crew they had doing it. There are a lot of things you have to consider. It’s “buyer beware” I think with 3D printers, whether you’re buying for yourself or your school. You have to really do your homework from many different resources. I would read as many different reviews as you can. I would go on also to the knowledge base, users support forums. Most of these manufacturers will have a forum where users are talking about their experience and see what other people are saying who have used the printer, what kinds of things they’re printing on them. At some point, you’ve got to pull the trigger. You can’t research forever. It’s like researching what laptop to buy or what computer. I guess it used to be maybe more critical ten or twenty years ago when there was always a faster chip coming out next month. If you bought that important, really expensive computer now, next month there will be something faster, your value of years will drop like a rock. If you wait forever, you’ll never get one. At some point, you’ve got to jump in. Getting some experience is better than none.
The other thing that I have an issue with this is the tight rating. You either get the score or you don’t get the score. The span of ratings from a 45-point scale, because there’s nine things that they’re rating against, five points each. Out of a 45-point scale, the best one did 36 and the worst one did 20. Most of them did around 30 to 36. That’s pretty tight. That’s not a big span. Interestingly, the Robo 3D got a twenty, the old Robo. They got the worst. Robo 3D, that printer’s been around for many years. Of course, Robo 3D has just come out with their new lineup. I don’t think it was done in time for the Printer Shootout because they’re just coming out here in 2017 instead of 2016. Maybe they’ll be reviewed next year. I think that probably is more an indication that that printer is several years old and a lot of these printers are newer and they have more capabilities. Things have refined over that time quite a bit since that printer was made.
The other thing that I have an issue with this is the tight rating. You either get the score or you don’t get the score.
I also think that for the bulk of the printers having a six point swing between them is not very much. I don’t think that it is really descriptive of the significant differences between a lot of these 3D printers. Here are some of the criteria that we like to really evaluate and what makes a printer standout in our minds. This is how we go about it. We really probably should be writing a blog post on more about our test criteria and how we do that. I think that’s something we should make happen in the next 30 days or so.
We really look at, is this easy to maintain? Is it easy to set up and get started? Does it do really pretty decent quality for what you’re trying to print just right out of the box? We don’t want you to spend a lot of time figuring out how to get it right, how to get it calibrated, how to get it going. All of those things just frustrate you in the process and seem like make you feel like you’re wasting money. We want you to feel successful as soon as possible because then you will dive in deeper. You’ll learn more, you become more expert at it. Then, any of those challenges that come your way will be fine. You’ll be more capable and able to handle them than you are at the get-go. This is something that we really look for in the printer.
But as you get to be pro, we also look for the same things. That’s what’s so interesting that we found out over time, because who wants to mess with that junk? Who wants to deal with the hassles of constant filament jams or recalibrations of the bed? Those types of things over time just make us like it less too and want to seek a new machine that’s simpler. It’s interesting that in the beginning, we want to spend less time dealing with the machine and more time learning how to print. Then, once we learn how to print, we want to spend more time learning how to design and doing the design work than we do want to deal with the printer. All in all, it has to have the simplicity for what we want to do and what we look for and what we think you, giving us feedback for, that that’s what you’re looking for.
Another thing that’s an important criteria that really isn’t covered, at least consistently. I think there were a couple of printers that this was mentioned on, but again, it’s not something that they rate overall across all the printers. How much time are you likely to spend maintaining the machines versus how much time it’s spending making useful parts for you? Sometimes a more expensive machine that’s easier to maintain and use and has maybe auto-calibration is more reliable, even though it’s more expensive.
As an individual or maybe as a small business, if you’re spending less time messing with the machine just getting it to print things properly, then even though that’s a more expensive machine when you buy it, down the road it may be less expensive because you’re using less of your time. You’re spending more of your time either finding or building in CAD what it is you want to make than you are maintaining a machine. Maybe there are other people that are wanting to learn more about machines, what makes them tick. You might be a person that would like to have a machine that’s either a kit or hackable or something that you can modify and make more what you want it to be. That’s an education you’re trying to get because that’s maybe more in your field or your interest area. Maybe you want to push the edge of the envelope and create something new about a 3D printer yourself. Maybe you’re into that. That type of issue is not really covered at least consistently.
Then two other things that I find missing in this, one is that, they talk about it on the video on their site. You can see some of these little prints, the types of models that they’re using. They show you what they are. What print is demonstrating overhangs or bridging or whatever. They have examples of them and they show them on the video, but they’re not showing you the one related to a particular printer. When they’re showing you those things, they also say that they’re doing one overnight print, which probably is testing, does the machine run all night? Does it crash during the print? What happens with all of that? It’s more about the reliability. I don’t see any ratings or rankings or any information on that.
I also love to know the time to print. I think it would be good to have a certain test print that was just for a speed test. This isn’t saying you want to print it fast. We’re saying, “Hey, you’ve got a significant print. On this printer it’s a six-hour print, on this printer it’s a four-hour print.” Maybe speed is really important to you. Especially if you’re going to use it in a pro-situation which you’re going to be making these parts, like if you were manufacturing or doing some on-demand manufacturing of something in a craft situation or whatever that is. We discovered that from one model to the other on the MakerBot when we got the new model to test out. The printer speeds went up because it’s a combination of the software and hardware that made it better. We’re like, “Wow, great. The same print that used to take us 20 hours is now taking one third faster really, which was amazing. From a productivity standpoint, if that matters to you, that’s a good reason to choose the printer.
Although then again, I’ve always been wanting to sacrifice speed for quality in a lot of things that we do. In principle overall, I’ve been wanting to prefer quality over speed. I think there’s a lot of other parts that are just not aesthetic parts, more functional parts, maybe more engineering parts where the surface quality isn’t as important as long as the part will function as it’s needed. There you want speed over quality. There isn’t that kind of judgment or comparison either. If you’re doing this overnight print and the overnight print takes twelve hours, and it takes only ten hours on one printer but the quality of the print looks the same from one to the other, then you’re not sacrificing that. Being able to see that and have a comparison. Quality and speed, happening both things, but being able to look at those objects and go, “Wow, they look exactly the same. It printed perfectly on each one. This one took a third less time or 20% less time,” whatever it might be.
These are really important considerations. I think, it’s a real difference between a technical perspective of capabilities, features and benefits in a finite sense, to a user experience perspective. It’s very different. It’s something that we find. We actually have had a couple of 3D printer companies come to us and have us test out their new printer before it’s released because they want a user perspective. All of their people internally are focused on the technical aspects of getting things done, getting them done on time. They don’t have that user perspective. In a nutshell, what we’re really talking about in general here is we have a preference for, and I think a lot of people that ask us, “What 3D printer should I buy?” are coming from a user perspective on that, than they are a technical perspective. If you line up printers, their ratings side by side with this number point guide system, that’s one way to look at it. But it may be doing users a disservice to not be considering some of these more real world issues instead of the technical ones.
Printer Shootout testing team.
I long for something that we used to have a lot when we were kids. Did you know that Tom’s grandfather’s name was Hap Hazzard? That was his nickname and everybody called him that. He had it on his business card. It was embroidered on some of his shirts too. Everybody called him that, including us grandchildren. Everybody called him, “Hap.” Hap was a fanatic for the consumer reports. If he had to buy something, he would flip through and look at that consumer reports. Pulling out that consumer reports magazine and looking for how they were rating the latest TV model, the latest radio, whatever it was that you were out there to buy made a big different because it was always from a consumer’s perspective. They had the value in that. Make Magazine, you should always maybe be from the maker’s perspective, but it’s harder because not all makers are alike.
To be fair to Make Magazine, they clearly are supporting the makers out there that are generally more technically savvy. That’s fair. That’s how they’re leaning as they set up their rating systems and conduct these reviews. I think a lot of other people go and buy these magazine only who aren’t subscribed all year long to all the other things we’re talking about for makers just to get the printer review, to help decide what might they consider buying or what might they want for their schools or their homes. You can understand why they do what they’re doing. Going through this, it raised more questions than answered a lot of the times.
This is just our two cents on this. It’s not meant to be a rant. I know it’s easy to criticize. Anybody from Make Magazine takes issue with it, reach out to us and we’d love to have you on for an interview and talk more in-depth about it. That would be great. Hopefully, we’re just trying to help those of you, because we get a lot of questions frequently from new users. I think it’s important to just consider some alternative perspectives. We know that their rating of the year’s Best Printer has a direct affect. We see it all the time. We see it in the number of 3D Hubs that are running this printer and the stats that come out. There is a direct result on the sales of these printers. Yet, they don’t seem to last from year to year. It doesn’t hold up its trend as still being one of the best printers out there. That I think says a lot about whether or not that’s just, “Wow, I found it. I liked it, but it’s not lasting for me.” You have to really take and weigh that. You’re going to get it in, you’re going to be frustrated with it, and you’re going to eBay it. You want to make sure that you’re really going to get it in and you’re going to keep using it. It’s going to do what you want it to do.
It also calls into question, and this is if I was to talk to the people in-charge of this at Make Magazine, I’d want to know, how did you determine this year over last year what 3D printers you’re going to review? They really don’t talk about how they determine which ones they were going to review. Consistency year over year, why was Robo R1 in there, that is a printer that’s five years old at least, versus ones that are brand new? How are those decisions made? That would be a really interesting question. It raises a lot of questions.
Anyway, we have a lot of printer reviews on 3DStartPoint.com. We have quite a few that we’ve done over the last year. We’re trying to do one a month going forward at minimum. We have a couple more that are in the process of being tested and writing right now. That’s been a little while since we posted one and it’s because some of these just have required a little more time, but they’re coming very soon.
We also are going to be pushing out and talking about repurposing those reviews in a way, so that you can get to see which ones might have some education applications. We’re going to re-point to sections of them. You’ll be seeing more and more blog posts on that. We won’t be doing a podcasts on them, but we’ll be doing blog posts on them as the year goes through, compilations. You should be able to walk through and go straight to the reviews that maybe relate to the best educational 3D printers or 3D printers that are for this particular application or use or the easiest to maintain 3D printers. We’ve been doing a couple of those blog posts already. You may not realize that if you haven’t been to the site recently. Check out 3DStartPoint.com and check out our blog post.
One little note about those reviews, sometimes people ask, “Why haven’t you reviewed this printer or reviewed that printer?” There are a lot of factors that go into that for us. Usually, the reason we’re reviewing a printer or not reviewing a printer has to do more with the manufacturer than it does us. Some of them are very happy to send in a printer to be reviewed, others don’t. We’ve had one company for more than a year telling us, “We’re sending you a printer for a review.” We’ve been expecting it, waiting for it, it doesn’t come. We review as many as we can, as many as we can get our hands on. It’s just that there’s no consistency in the circumstances. We’re trying to do as many as we can.
Once again, as I was saying about 3DStartPoint.com, you can go there on the blog post, but if you want to be notified of new blog posts, not just the new podcasts going out, put your email address in there and you’ll get a ping notification. You can set it for weekly, if you only want to get a weekly of how many blog posts and podcasts were out this week. You can choose weekly or daily. That’s right at the top of the Home Page, right where the moving 3D print ties are. If you’ve sat on the Home Page for a couple of minutes, you’ll see it’s an animation of sorts. You won’t get spammed for us or sell our list. We don’t really even like email lists, but this is very useful for those of you who would just prefer to know when the blog post is posted. Then you’ll know when there’s something new that’s worth coming back to check out.
You can go on social media, @3DStartPoint, specifically in Facebook and Pinterest. You can find us there as well. We usually always push out a message about the latest blog post, as well as the latest podcast.
We will be back tomorrow with another episode. This has been Tom and Tracy on the WTFFF 3D Printing podcast.
Important Links
Listen | Download | View
Hear the episode of the WTFFF?! Podcast by using the player above OR click to download any episode.
Help Us Help You!
Have some feedback? Leave a comment below. We will read and respond.
Please also review us on iTunes and share via the social media of your choice.
- 3DStartpoint.com
- 3D Startpoint Facebook
- 3D Startpoint LinkedIn
- Hazz Design Twitter
- 3D Startpoint YouTube