Reading about MNLARS – the state’s drivers license, registration and titling system – is making me nostalgic.
Back in 1996-97, I worked for a company that was engaged to do the engineering for a big, extremely well-funded startup in Palo Alto, run by a former salesman from IBM. The company’s business model: pay people to read spam. The theory was, people would set up accounts, and then get a little bit of the ad buy money for clicking on ads, and links in spam emails.
Catering to greed, of course, is never a bad business model. But by the time I left the company, we were figuring that someone clicking on six ads a minute, 8 hours a day, might make $6-8.
I wound up leaving – and was delighted to read in PC Magazine that the project had made their “Ten Dumbest Ideas of 2007” poll. By this time, I the company had folded, taking $30 million in investor funding with it (along with, I was less delighted to learn, the company that I’d worked for).
Remember when $30 million on a bad idea seemed like a lot of money?
If that’d been a government project…
Oh, wait.
The Minnesota taxpayer has spent close to $100 million on the MNLARS system – half up front, leading to a spectacular failure, and half for a “fix” that failed even worse.
And now Governor Walz wants to double down on the failure:
Walz released a budget this week that includes $94 million through 2021 to finish the system known as MNLARS, operate it for two years, hire staff, and reimburse deputy registrars who took a financial hit from the botched 2017 rollout. That’s on top of $15.7 million in stopgap funding that Walz was already seeking to get the system through June 30.
Via Fox9
To pay for some of the costs, Walz has proposed a $2 fee every time a driver makes a vehicle license, tab or title transaction.
Fearless prediction; we’ll end up spending $400 million, and end up doing the whole thing in Google Drive spreadsheets.
If the Titanic was MNLARS, it would have hit the iceberg at noon on a clear day.
I usually don’t get angry about failed projects; annoyed, sarcastic, cynical, yes, but not angry, but I’m gonna make an exception with MNLARS.
Above all, the most important takeaway is that the state replaced a working system with one that doesn’t. Or barely. And can’t be fixed. Hardly. And needs still more money. With no end.
The whole project violated everything I know about project management. There should’ve been working, functional, focused tests – while the old system continued to run, so the transition would be as seamless as possible.
Moreover, I haven’t heard of anyone who was fired or laid-off for this fiasco.
Not quite.
The old DMV system had to be replaced to comply with Federal Real ID requirements. The state was forced tp get multiple extensions from the feds so that Minnesotan could use their driver’s licence and state id’s to board an airplane. They were on their last extension.
Beyond that, critical portions of the code were written in BAL (hey IT guys,how is that for a blast from the past?) thus tying the system to IBM mainframe systems, an architecture that IBM very much wants to sunset. The way that IBM forces you off an architecture is to ramp the maintenance fees to well beyond the point of pain.
I read the auditor’s report and yes, there were serious project flaws but the critical error was contracting with Hewitt-Packard, who just plain screwed the state of Minnesota and burned precious time (years actually).
Beyond that, there were typical errors.
– Subject matter experts who were required to keep up with their responsibilities AND devote sufficient time to MNLARS
– Failure to modernize business processes before automating them (some of this goes back to time lost with Hewitt-Packard).
– Insufficient personal assigned to testing.
– Failure to properly regression test.
– Hiring a project manager from Wells Fargo who did not know the business, nor the people – but thought she could manage it anyway.
But to be honest, every project that I have been involved with in over thirty years of IT has suffered from these same flaws and still succeeded.
Paul Meekin was terminated last April. Susan Rohde “resigned”.
I’ve seen similar system development failures in the private sector, though for far less money ($48-$52 million), that resulted in 38 people being walked out the door one glum October day. The rest of us mice stayed real quiet for a couple weeks.
One more thought. Paul Meekin was the sacrificial goat. He knew he was over his head and made it clear to everyone that he was. He was handling day to day operations at the Department of Correction, the Bureau of Criminal Apprehension and Drivers Services as well as handling MNLARS.
If one is looking for a villain here, it is in the Governor’s office and the state’s business practices. Try writing a freak’n state contract sometime! Or just go through the dysfunctional process and you will begin to understand MNLARS.
Most of the MNLARS effort, 3-4 years actually, were wasted just trying to write a contract with HP.
It’s nuts.
The state needs to reform this, but reforming this don’t put money in the DFL’s pocket.
Well I tellya… I was on sight as a contractor in very recent years and they were doing their functional reqs and tests on spreadsheets. The best of 97 technology.
^ I can’t believe HP wouldn’t have delivered a functional product for a price. And that’s not to say one should think a large enterprise like HP is the be all end all… But that’s what they do.
^ Greg – I think that’s true, Meekin is just the guy who was holding the bag by virtue of having tried to do an impossible job.
MacArthur, I totally agree. I have watched massive projects go down in flames, all while passengers were screaming at the frozen in panic pilot that the plane was going down.
In my opinion, the best IT story ever written was an oldie but goodie written twenty years ago. What Lou Gerstner Could Teach Bill Clinton
Here are some pullout quotes:
And here is the big one
There’s another thing going on at MnIT that won’t / can’t be articulated anywhere, and its about as big a problem as anything else. It’s an H1B ghetto over there. And I say that having real respect for the south asian SW engineers one meets in the broader field. But at the state the staff is almost all foreign and very young. There’s almost no staff with meaningful experience in US commercial or government enterprises, nobody there who has an intuition of how things should workflow. The state is being taken advantage of by the staff augmentation vendors, but its not as if there’s another way to staff the projects the way state employment works. State wont pay market rates for domestic software engineers, either FTE or contract.
The question is, would it have functioned as the state’s DVS system? A functional product is not necessarily a functioning product.
I worked for Mortension Construction for a couple of years. After the first week working with the bid group, I asked a very naive question at lunch.
“How do you guys make any money? I mean…..your bid only accounts of less than a 1% profit and given the cost of money, what gives.”
I knew it was a stupid question but didn’t realize how stupid until I saw the smiles around the table.
One guy was kind enough to explain, “No one EVER wants us to build what they asked us to bid on. There are always changes….and every change, costs and costs and costs and costs and in the end, we make a bundle.”
And that is what MNLARS was running into with HP. They saw requirements as clearly defined in the contract and HP saw them as change requests.
It got dysfunctional fast.
Couldn’t agree more….but the problem is worse than that. State IT managers suffer from a cult of management problem whereby they believe that management practices and methodologies can make up for not knowing the people nor the problem, which leads them to blatantly disregard the few people who have any expertise.
I have seen cases where an entire project was let out for bid without the knowledge or involvement of anyone in the department below the rank of manager.
That is a level of hubris that exceeds the ability of the English language to describe.
Not quite
I accept that the system was probably non-compliant with regards to any requirements, but it worked for the purposes of license plates and tabs. That these aspects should’ve been involved in the fiasco is unforgivable.
So, are our driver’s licenses Real ID compliant? Is there anything even slightly good in this?
Two people paid some sort of price for a $100M failure. Big deal.
Sorry: “any Real ID or other federal government requirements”.
Paul Meekin was terminated last April.
He didn’t go away quietly: https://www.mprnews.org/story/2019/02/15/mn-paid-ex-it-official-ousted-over-mnlars-not-to-sue
Because no one ever gets fired from government. And when they do, time to pay the piper.
Thanks. Thanks a whole metric crapton, public sector unions.
A guy like Meekin didn’t have any meaningful union protection, he was management.
I don’t blame him for putting the screws to them such that he could…. he was scapegoated.
Scapegoated tho’ he may have been, was he innocent? I think a $45k settlement indicates no. And all concerned are wiping off the brow sweat because a full-on lawsuit would’ve allowed discovery which the state of MN seems to not have wanted.
Innocence is not the right word, that’s legal / criminal stuff.
He felt he had a wrongful termination / defamation claim that he could credibly pursuit in court. And he did, which is why the state settled… it was half a dozen either way, spend tens of thousands settling now or spend tens of thousands litigating and maybe paying a claim also. Thing is Meekin had the leverage of a credible claim to pursuit.
When I read the auditors reports, what I see is the mass psychosis of these state managers all green lighting on the go / no go decision. What could Meekin do? He had no reasonable basis to hold up the roll out what with everyone saying it was good. Obviously, he had a weakness in not being a big enough personality to stand in front of the tank there. But that also would have put his career at the state in jeopardy. It was kobayashi maru, not a winnable situation to be in.
There’s not a lot that discovery could have done for Meekin that that auditors reports didn’t do already for the court case. The auditors reports were public and supported quite a bit of what Meekin said in his own defense.
From 1990-1999, I worked for an IT staffing and consulting company. Anderson Consulting, now Accenture, had a reputation for padding their project costs by bringing newbies in from other offices outside of the state, that knew nothing that would help on the project, but Anderson would train those people on their clients dime. I had two top flight process guys that helped me narc Anderson out on a project at ING. The project sponsor asked us to bid it from that point to production. They booted Anderson and gave us the gig. We finished the project ahead of schedule and under budget. From that time on, I ate Anderson’s lunch on about 10 projects.
A few years later, I was working for a low voltage cabling contractor. Our PMs were so savvy that they anticipated hidden costs that ordinarily would generate a change order, so our bids usually came in higher that the companies that low balled the projects to get the bids, then used change orders to get more money. I turned a few of my customers from competitors when I pointed that out.
My point here is that HP did the same thing and our new AG should be spending his time going after them instead of his useless suits against Trump.
I know that we’re all small govt here and that informs our worldview but I’m not sure that the “New MNLARS” shouldn’t cost $100m to do it right in the first place, and that the problem is thus that the state gets sucked into nickel and diming it because of the political budgetary culture at the capital…. ending up wasting $50m here, $50m there with nothing to show for it.
MNit’s salaries are not competitive with the commercial market, they want their top of the bracket to be like $48/hr for java guys. They dont get people who can do the job at that $.
Ding! Ding! Ding!
“What’s that sound?”
“Staff augmentation.”
“Huh?”
“It’s the sound of our consulting company backing a school bus up to the door.”
^Greg “Hiring a project manager from Wells Fargo who did not know the business, nor the people – but thought she could manage it anyway. ”
My insight is what I read on LinkedIn, but Rohde was an SVP at WF for over 10 years. There’s got to be a story for why you go from that to making a pedestrian $150k at MnIT.
John,
WF has been busy tarnishing its brand for the last 11 years, it would be worth taking a more granular look at her resume. From my time there, VPs are literally a dime a dozen at WF (a “bank officer” being necessary to sign routine paperwork), SVPs are still just middle management, the real decision makers are EVPs. I would look at pairing her tenure with some of the unfavorable incidents that made it into the business press in the last 11 years. The other thing I would look at is how many projects she worked on that actually made it into production. WF starts and kills a lot of projects.
In a former life, back in the before days, I built control systems for municipal water and wastewater treatment plants. Our contracts *always* had performance milestones backed up with financial penalties. There were also always points at which the plug would be pulled, too, and we would end up holding the cord if that happened (although it never did while I was with them).
So, I’m always at a loss to understand how these projects get so effed up, and why the taxpayers always end up paying for it. Maybe it’s Minnesota nice…I dunno.
Swiftee,
Great points.
There is a large, privately held company here that was going to implement Oracle. Once the project exceeded the contracted amount, which if I recall, was over by close to $3 million, they pulled the plug and booted the implementation company out. This was a few years ago, so they may have taken another shot at it.
John, just because I prefer small(er) government, doesn’t mean I find the idea of paying $100M to be too much for a functioning system. It’s a bit insulting. My problem was and still is that I have no indications that there won’t be another failure and another until we hit that $400M mentioned by our host.
These sums of money are huge. Really good programmers, system designers, etc can be hired for these amounts. The fact that MNit pays poorly is a lousy excuse; as is the apparent fact from the info above than MNit employs lousy managers. I’ll bet they have all sorts of diversity training and sexual awareness training and plenty of incentives to not do their job.
It really seems to me that this is all just lefty corruption (feeding at the trough).