Tuesday, July 5, 2016

My Year In Point-of-Sale Purgatory

Slightly more than a year after the process began, DSG is running a new point-of-sale system at more or less full strength.  There are some items and categories still being sold by manual line item, such as BattleFoam, because they don't appear in the new system's integrated catalog and we haven't gotten to them on the list of product cards to create/import.  But most things are working properly.

In the year this took, I spent over ten thousand dollars, much of it non-recoupable, and lost an incalculable amount of business due to infrastructure inefficiency.  Words alone do not fully express how frustrating, wasteful, slow, expensive, and damaging this process was to my business.  In the end, one of the devils I knew showed up with a nice new car and promised things would be better this time, so I went ahead and got in bed with him.

DSG opened in 2012 running Light Speed Retail's Pro (client) software.  LSR legitimately served the company well for a couple of years before entropy set in.
LSR Pro is client software that uses a FileMaker Pro database on a server Mac running OS X to provide point-of-sale access at terminals on the same local network.  Each terminal "lane" requires a license, but the licenses are not device-locked.  Thus, if a computer failed, I could whip out the iPad and log into LSR and continue doing business in that lane.  We started with two lanes (front and back of house), and expanded to three a year later.

One thing to make clear about Light Speed was that it lived up to its name.  The client application, especially following a lookup engine revision, was extremely fast even on slow hardware.  (Presuming my server was fast, I imagine, which an i7 quad-core CPU with a pile of RAM and an SSD generally is.)  It was also reasonably easy to use in most respects.  The database interface tools and utilities were generally robust, intuitive, and refined.  In particular, I think LSR Pro has the best purchase-order automation of any software I have run or considered running.

So I ran LSR Pro, and for a while, it was good.  And then XSilva, the publisher of Light Speed, acquired MerchantOS and decided to develop it into a cloud version of LSR.  The cloud deployment was not directly compatible with LSR Pro, and there was no migration tool.  (And there still is not one to this day.)  For the kind of insane bank they charge for service, starting at $80 per month per lane per login, one would expect fully turn-key everything.  LSR Cloud, while promising in many respects, is just not cost-comparable to its competitors in this space.

So what? you might ask.  Aren't you on LSR Pro?  Well, yes.  But the emergence of LSR Cloud and its subscription economy meant that XSilva effectively abandoned the client-based Pro, which they rechristened LSR Onsite in order to remind us Luddites still running it that we were tethered to the corporeal computers in our businesses like a telephone switchboard operator of old.
Gradually, things started breaking.  The Amazon integration started failing more often than before, I expect because of deprioritization of the software team's work on maintaining connectivity with LSR Onsite.  The LSR Onsite e-commerce module, something of an artifact now that LSR Cloud came built with such utility from the outset, broke entirely for me when they moved to a new version, and no reversion was available.  I found myself unable to take payments without an Error 500 occurring about half the time.  When people can't, you know, buy things on your retail website, that tends to constitute a problem.  Especially when I had already abandoned an existing web sales infrastructure running Crystal Commerce to sell via TCGPlayer Direct, eBay, and Amazon in favor of doing it all in a single, unified LSR deployment that was, briefly, far more efficient.

Worse than the breaking items was the lack of progress on known issues from before.  It became pretty obvious that XSilva just didn't care anymore.  Owing invoices still generated far too easily and were non-intuitive to hunt down and eliminate.  The kernel crashed about 50% of the time that I ran sales metric reports from the "Intelligence" panel.  Any loss of connectivity between the server and its clients would take a given login lane key out of service until the server was manually "stopped" and restarted in its system settings pane.  A number of glitches when adding and subtracting discounted items and changing prices on invoices from the browser meant an unacceptable amount of "just make a new invoice" workarounds.  Taking and counting inventory through the existing interface was a huge chore.  And, perhaps worst of all, there have been problems since day zero with the EMV integration through LSR's exclusive in-house processor, Cayan, and some of them remain unsolved to this day, almost a year later.

By early summer 2015, it was clear I needed to leave Light Speed, but I didn't have a solid place to go.  I began extensive research into options.  First of all, to keep online operations running profitably, I went monochannel eBay.  For all its faults, eBay has served me dependably for sixteen years.  My account, now the store's account, has thousands upon thousands of feedback and is in absolute God mode with everything unlocked.  I could post a pre-order listing for an aircraft carrier, take payment, and withdraw the cash immediately.  I can list in any category with no restrictions.  I can list one of everything in my store and not run out of allowable items.  Anyone who has tried to develop an eBay account from scratch since 2014 or so knows the power of which I speak.  It's heady stuff.
The problem with eBay, of course, aside from the fact that you're chasing what is often a pretty narrow margin, is that there is limited efficiency without an integration platform to the rest of the POS system.  Even an expert who lists quickly and knowledgeably and uses Turbo Lister and every other tool at the monochannel eBay vendor's disposal is going to be left in the dust by software that does the work by itself.  This was a band-aid, a tide-me-over, not a real solution.

I approached my POS system decision with a list of priorities.  They went, roughly:

  • The system needed to have better inventory processing options for comics than LSR did.
  • Customization was a dirty word; I wanted off-the-shelf standard modules to the extent possible.
  • Device-agnostic operation was preferred.
  • Online integration was no longer optional, but mandatory.
  • The system had to work at scale; my three LSR lanes were already proving too few.
  • If at all possible the system had to work remotely for convention sales.
  • Cloud deployment was acceptable if I gained real ease-of-use out of my trade-off of giving up the lightning speed and tight security of a client-and-server local network deployment.

I journeyed a surprisingly short distance with so specific a filter set in place.
I looked into Bitter End Systems' MOBY POS because of its deep comics integration on the OS X platform.  As a retailer already comfortable administering a network of Macs, this option appealed tremendously.  The disqualifier came not from MOBY itself, but because it requires a licensed copy of FileMaker Pro on the terminals, and beyond the first it starts getting hideously expensive.  My FileMaker Pro running internal to Light Speed wouldn't be valid.  MOBY us also earthbound, being a client-and-server install.  I didn't write MOBY off completely but it jumped over to the "if I don't find a better option" column.
It only made sense for me to explore Intuit's QuickBooks POS, given how many of my fellow retailers are running it.  I knew it would work in a multi-store deployment because Josh Fohrman of Game On and Pat Fuge of Gnome Games are using it to do exactly that.  Even better, with my business back-end already on QuickBooks Online, using the online version of their POS had some very direct appeal.  I'm already deep into the QBO app library with things like AMEX syncing and Intuit payroll.  I smiled at the thought of how much accounting work might be automated away.  And it was a cloud system, so I would be untethered.

I was ready to make some serious inroads to QBPOS, but it disqualified itself.  Intuit was unable to deploy EMV compliance on time, and delayed their liability shift until when-the-hell-ever.  I discovered their inventory system was nowhere near at the level I needed, and their store credit handling left a lot to be desired, and worked best when appended to a gift card plug-in or like such.  I never like running something that depends on a bunch of plug-ins, extenders, bootloaders, or what have you.  Maybe it's part of my Apple workflow preference, I don't know.  But the inventory thing was the clincher, it looked like it was going to be a nightmare to try to handle comics on that system, let alone TCG singles.  Across the board I started seeing QBPOS look inadequate on my healthiest categories: TCGs, comics, and the soon-to-return video games, which had been healthy in their original tour of duty at DSG.  I slotted QBPOS below MOBY in the "if all else fails" column.
Microsoft Dynamics Retail Management System (RMS) is the gold standard in small retail and has been for a decade or more, and Diamond Comic Distributors sells RMS paired with its own companion app ComicSuite.  This was where things went expensively wrong.

See, ComicSuite RMS ostensibly hit most of my criteria.  Its inventory depth is staggering; the kind of power and control users have over inventory in the system already is without peer, making even Light Speed look weak by comparison, and making QuickBooks POS look like an absolute joke.  Its purchase order automation wasn't at LSR's level of ease, but was similarly versatile and powerful. ComicSuite downloaded my weekly Diamond order line item by item into RMS and set it up automagically, including reporting of damages and discrepancies.  The RMS back-end is similarly robust in terms of reporting, user access control, general security, record keeping, and compatibility with a vast array of plug-ins, well beyond what I'd ever want to use.  Best of all, people whose opinions I trusted ran RMS.

I took the plunge.  I spent just under $3k on a Windows 2012 blade server.  (It was supposed to be over $4k but I toned down its beef at the last minute, and I'm glad now that I did.)  I bought three new terminal PCs, because RMS is Windows-only.  I bought assorted monitors and peripherals for all, Symantec for all because Windows is still a filthy virus and malware haven, and a rack mount and power conditioner for the server rig.  For three lane licenses I ponied up just over a grand each.  All told, the RMS aspect of my POS migration by itself cost in excess of eight grand.

And then it didn't freaking work.

So, ComicSuite likes to misbehave under Windows 10, so I kept the terminal PCs on Windows 7 Professional.  Regardless, the periodic ComicSuite updates kept breaking the install and forcing me to set it up again.  I had a good friend help me set up the SQL database on the server, and in the end the stuff he worked on was about the only part that stood up.  The POS interface offered a paucity of functionality compared to other packages.  The import tools were, well, calling them "rudimentary" suggests that they worked to some modicum of decency, which would be overstatement.  From ComicSuite itself to the various database utilities, the glitch-ridden opacity abounded.  Any remote access depended on a combination of Remote Desktop and (for my own workstation) Back to My Mac, and those are each fussy in their own ways.  But the clincher, the thing that killed it for me, was learning that I was still nowhere near the integrative functionality that I needed, at a time when my TCG singles and video game businesses were exploding with forward momentum.  I was going to need months of work installing, learning, and deploying plug-ins just to get to where the cloud software started you off on day one.

And all of this was besides the misery of losing a bunch of non-POS functionality that we had become used to using in the business, as a result of migrating from OS X to Windows, and being hidebound to a client-and-server physical IT deployment in the store of significant extent and creating significant failure vectors.  Worse still, lane licenses were locked to their respective computers, at least requiring a transfer process in the event of failure, rather than "log out of the PC and log in on the other computer or tablet" flexibility.  It seemed like the people who had the best results from RMS were those who had already been running it for a long time, and whose hardware base was already cultivated to the task.  I set out to mimic the best deployment I had encountered for RMS, only to learn that it does not cooperate on that level fresh from the box.

Much of the problem was ComicSuite, and I could probably have made RMS work for me in its core form, but the other POS options overtook RMS in that comparison.  Furthermore, the software was already end-of-lifed by then, and is set to migrate... eventually... to Retail Essentials Hero.  RMS is deeply powerful internally.  I think its Essentials Hero evolution will probably be appropriate for comic stores not dealing seriously in games.

I had bought the hardware for RMS in November and December 2015, and then bought the software in late January and started the setup.  In the meanwhile I had been following the progress of a new entrant into fray: the Inventory Management Program POS, or "IMP."
IMP became a reality in December 2015 when Backstage Hobbies owner Nate Petersen developed a point-of-sale front-end and a set of API pricing tools for TCGs and video games and bolted them onto the robust OpenCart application core.  When it became clear in roughly April that ComicSuite RMS wasn't going to get it done, I bought in on IMP with the expectation to run all operations on it.  The setup price was reasonable given the risk, so even though it's part of the unrecoupable portion of that ten grand, I don't consider it to have been bad judgment to try.

Like RMS, IMP was built on the SQL database platform, and this time the inventory management tools stood up well, while the system's overall durability and scalability weren't in the range I needed yet.  At this stage IMP is the only POS system I know of that has live pricing from Video Game Price Charting for that product category.  For software seven months old and still very much in beta, IMP shows promise, and is already a feasible option for deployments in owner-operated stores with few or one register lane, or as a web-facing store for singles.  Unfortunately, I did not have the luxury of time to let the software catch up to business.

My several weeks attempting to deploy IMP were beneficial in more ways.  I realized I would want to keep the PCs I had bought to use as RMS registers under whatever I ended up deploying, and counting monitors, scanners, printers, and cash drawers, that ended up making about $2400 of my ten grand an effective recoup.  My old iMacs served DSG well for years, but they stagger forward now under the weight of seven years of heavy usage each.   When I sell the RMS server and bring in (for presumably an aggressive sale) the gratis back-catalog trade paperback package that Diamond affords its ComicSuite subscribers, I'll be in a position where the direct costs, at least, have been recouped down to a bottom-line loss of only around $5k.  And if I can get any kind of value out of the "working copy of RMS" aspect of that server, never mind that it's not nominally transferable, then the recoupment becomes better still.  This does not take into account the money lost to business infrastructure deficiency, but as I've said, in business you don't have what you can't count.  That figure will remain forever speculative.  It stings all the same, despite being rooted in the sunk-cost fallacy.

I also learned during this time period that TCG singles, as a segment of my market and sales, had become utterly dominant.  The increase curve on singles defied all logic and expectation.  If I had floated my April sales past you gratuitously and told you to look at how much of our ~$100k month came from Magic singles, you wouldn't believe it, you would think I was making the figures up.  And this, in a store where I focused on diversity and giving other games the attention that most card-focused stores give singles!  But no.  Despite my comparative indifference, singles refused to bow to impediment.  They would not relinquish attention.  They dominated sales, then the buy budget, then labor time.  And when singles are the soup of the day, there is one POS software package that stands above the rest.
And now here we are.  Dan McCarty's Crystal Commerce, the much-hated cloud SQL tool suite that integrates coarsely but functionally with TCGPlayer, eBay, and Amazon, that has outstanding function problems going back years that its development staff seem to be in no hurry to correct.  Crystal Commerce, that had become a Spongebob meme among retailers as a result of that same developmental inertia.  Crystal Commerce, that I had discarded in late 2014 in favor of going all-LSR for lack of any divination that XSilva would lose interest in the Light Speed client version within a matter of mere months.  Here I was, crawling back.

Sigh.  This is business.  All that matters is what is necessary now.  My friend Scott Church, who had continued to run Crystal Commerce for his Category One Games webstore, let me tool around town in his system and stress-test it based on where I had run IMP's wheels off the axles.  How did it fare?  It basically stood up to all the bottom-line daily functionality I needed.  Moreover, it looked like it would meet my migration criteria more closely than the other options.  Let's look at the list.

Comics.  Does it handle comics better than Light Speed?  Yes.  It does.  Better than ComicSuite RMS?  Well, if you can't get RMS to work, I suppose so, but its comics handling is nowhere near what a working ComicSuite or MOBY can do for subscription management.  Be that as it may, my Media Manager is adept at subscription upkeep using standard tools like Excel and a telephone.  Crystal Commerce had the one-up in the comics category by having a respectable depth of master catalog support for integrated sales of issues.  We'll be delving into that soon.

Is it off-the-shelf, requiring little customization?  Yes and no.  A SQL database is about as standard as they come these days, and Crystal Commerce's SQL tools seem better tailored to this industry than what RMS offers.  Compared to IMP, it's Playstation and XBox, you could make an argument either way, but the bottom line is it works.  The extent of customization is handled by the vendor, and there's a certain amount of caution to the wind that if your POS is down, so are many of your competitors.

Is it device-agnostic?  Yes, moreso than anything else available right now.  Unlimited lanes, runs on any browser, runs on a computer, tablet, phone, whatever.  Technically it should be possible to run this on a Nintendo Wii using the Opera Channel.

Online integration?  That's the core application of Crystal Commerce.  I won't be returning to Amazon SellerCentral, but I'll plug in our eBay account once most regular systems are working as planned, and I'll take another look at TCGPlayer when the coast looks clear.

Does it work at scale?  Yes -- so far any number of lanes operating simultaneously has presented no significant problems, working at home has not been a problem, and I have been able to deploy an entire side wall full of registers to serve customer traffic.  It even supports kiosk mode.  Moreover, CC is currently best-in-breed at managing a gigantic TCG singles inventory effectively, quickly, and with delegation to proficient employees.  The one scale logjam I found is that only one terminal can be importing products from the hive catalog at a time.

Convention sales are going to depend on what kind of data connection we can get, as it is typically not cheap to have an ethernet link, and wireless tends to be saturated out.  Assuming we can surmount those hardware issues, it appears trivial to run this software on a notebook while at remote events.

And finally, cloud deployment brought with it the revival of my store website, which I wrote about a few months back.  It looks considerably better now, and actually works.  (For over nine hundred dollars in "design fees" it had better.)  Event registration is a thing again, right from my customers' phones, as they demand it.  All pages beyond the front landing page feature feeds from our Google event calendar and Facebook page.  It passes the key killer app test: Are my customers able to buy Magic cards from their phones, pay in store credit, and pick up their cards before the tournament?  Yes, to every part of that equation.

There are certainly things wrong with Crystal Commerce even still.  The import tool and reporting tools are still extremely sparse and non-intuitive.  They promise new versions are on the way, and I'll believe that on arrival.  Actually creating a restock report workflow is needlessly convoluted and is one aspect in which CC is a mile behind LSR and a continent behind RMS.  There are a non-trivial number of instances of what we call the red-and-white "screw you" screen (the term is harsher than that) where the browser blanks and offers an opaque error message in vivid crimson and nothing more to go on.  Mostly, the Chris O'Dowd solution works, though: Turn it off and then on again.  A refresh might do it, a browser quit and reload might do it.  And the integration with TCGPlayer, eBay, and Amazon, is far from seamless and results in regular oversells even for careful retailers.  I'll enjoy having that stuff turned off for just a while longer.

The bottom line, however, is that Crystal Commerce works now, and works the way my consumers are demanding that my point-of-sale solution work.  It also works for the way I need my employees to be able to work, and they really like the fact that so much of singles transactions (in both directions) are cleanly systematic in the software.  The CC hive catalog, or integrated cloud product database if you like, reaches better than 95% of the products I carry, and I can create product cards for the rest, suboptimal as that is.

The greatest relief is finally being out of the purgatory position.  Since closing the store for one day on June 14th to change over the system and physically reconfigure our retailing infrastructure as well, I have spent three weeks making nonstop progress, each week better than the last in inventory coverage, system refinement, and ultimately sales.  Action items are finally disappearing from my desk again.  My staff knows what we're doing, they know why, and they have been able to focus on a vast, weeks-encompassing task without getting lost.  This is going to end up being a huge benefit to their later careers, having had this perspective from working on such a project.  It's one of those things you can't fully learn without doing it.

Crystal Commerce does get a cut of my online sales action on top of a subscription price that is not cheap.  But you know what?  I tried brewing my own.  It set me back thousands of dollars.  You just finished reading the sordid tale.  For a long-term moonshot where I had my own vast server infrastructure and the space and budget to maintain it, sure, I'd have stuck with ComicSuite RMS or Essentials Hero and moved forward from there.  But in a business that consists largely of making hay while the sun shines, I found a vendor offering a turn-key hay baling system.  It will have to do.  Dan McCarty, the progenitor of Crystal Commerce, is going to pocket a bunch of my money in the years ahead.  He will have earned it.

1 comment:

  1. This was really helpful and valuable information. We're happy with Square at the moment, but when we're ready to ramp up our singles sales we'll know what to do.