Complaint
AUGUST 2017
Submitted to:
Angela MacKenzie
Table of Contents
1 BACKGROUND ................................................................................................................. 3
1.1 MISSED OPPORTUNITY ..........................................................................................................3
1.2 LEGITIMATE EXPECTATION ....................................................................................................3
1.3 THE RFSQ PROCESS ................................................................................................................5
1.4 REQUEST ...............................................................................................................................7
Confidentiality/Validity
This document has been prepared by Evident and it’s consortium members for the sole purpose and
exclusive use of the Province of Nova Scotia. Due to the confidential nature of the material in this
document, its contents should not be discussed with, or disclosed to, third parties without the prior
written consent of Mr. Alan Haaksma, Director of Business Development, Evident, at
alan.haaksma@evident.com or by phone at (905) 616-4072.
1 BACKGROUND
1.1 MISSED OPPORTUNITY
The OneNS Report sent a message to all Nova Scotians that if the Province were to thrive, everyone
would need to do things differently. The 2016 Speech from the Throne talked about leveraging
procurement for economic development and innovation. The OPOR initiative provides the perfect
opportunity to do just that.
“OPOR is neither a system, nor an IT project. It is a vision, strategy and culture for transforming
the delivery of care for Nova Scotians that is enabled by underlying IT systems including a core
clinical information system. It is called ‘One Person One Record’ to emphasize that the vision is
not just about the care of individuals as patients but more widely encompasses the overall health
and wellness of the population. This vision, “One Person-One Record,” will provide a foundation
for improved health system governance, integration of patient information across the care
continuum, and standardized processes and practices.”
By prematurely eliminating vendors prior to proper evaluation of the overall opportunity they present in
terms of strategic alignment to Provincial priorities, we believe the RFSQ process led to a missed
opportunity for Nova Scotia to maximize innovation, while greatly reducing project and adoption risk on
a generational ICT spend.
Over the last two years we have worked with the province, NSBI one of your representative agencies
with other local firms to build a world-class consortium that has been focused on a holistic response to
Nova Scotia‘s OPOR procurement. The Consortium is made up of local, national, and international
companies including world-class players, Hitachi Data Systems, Bell Canada, KPMG as well as strong local
partners MOBIA Technologies Innovations Inc. and DavisPier Consulting. The Consortium goal was to
address not only the vision of One Person One Record (OPOR), but also the economic development
goals that were outlined in the Ivany Report and further emphasized in Nova Scotia‘s Speech from the
Throne last year.
To date, the Consortium has invested over eight million dollars in our pursuit to ensure the technical
solution will meet the needs of the Nova Scotian’s for years to come. Our intent was to establish an
International Head Office in Halifax, to not only service the Nova Scotia healthcare system, but also to
serve future clients across Canada, Europe and the Caribbean. Part of our planned expansion into Nova
Scotia also includes a Digital Health Centre of Excellence to advance health research and planning, and
to collaborate directly with Nova Scotia Health and the local community to develop and enhance
Thrive’s current functionality, using NS as the lab to develop exportable IT/IP locally, as well as to
position as an anchor client in the health ICT sector in Canada.
1. Open for Business - the provinces Economic Development agency, NSBI convinced us that Nova
Scotia could serve as a perfect location for our first move outside the US. A large and diverse
culture supported by world class Research and Development with an aging population in need of
infrastructure improvements, we saw a cultural affinity with our head office city, Mobile AL.
Our company is based out of Mobile, AL and the similarities between Halifax and Mobile are
nothing short of astounding. Same population, same land size, both ocean-going ports, both
ship building towns, both cities settled by Acadians and most importantly our southern culture is
a perfect fit for Atlantic Canada. As one of the largest employers in Mobile, we were and are
still convinced Halifax is the right landing spot for our International Head Office.
2. Economic Development - NSBI stated at the start of our journey that any procurement this size
(OPOR) would require significant Economic Development for the province.
3. On the Mark -In the fall of 2016 the Evident consortium submitted an unsolicited proposal to
address the OPOR vision. The unsolicited proposal was rejected. On November 23 2016, the
consortium had a debrief call with Chris Mitchell, NS Chief Procurement Officer. Mr. Mitchell
noted that our proposal hit the mark and that we articulated very well the challenges that NS
faces. He also acknowledged that PWC confirmed any of the mainstream vendors could solve
NS technical challenges. He also strongly encouraged us to bid on the upcoming procurement
process and that he expected a higher value will be put on innovation and economic
development given the magnitude of the procurement.
4. Good Alignment - The OneNS Report sent a message to all Nova Scotians and to us, that if the
Province was to thrive, everyone would need to do things differently. We are very confident
that our Consortium understands and satisfies the objectives in the, “Now or Never Report” and
subsequent, “Action Plan”.
5. Good Alignment - Speech from the Throne, October 13th, 2016. One of the overarching themes
was for NS to leverage procurement to drive innovation and economic development.
The sheer magnitude of the OPOR procurement represents a once in a generation opportunity for Nova
Scotia to attract significant economic development. Any of the current vendors can solve the OPOR
technical challenges or they wouldn't be in business. Evident has been in business for almost 4 decades
and our system has been successfully deployed at over 650 hospitals with tens of thousands of end-
users. In comparison, our US footprint is larger than the entire Canadian market. We can handle the
province of Nova Scotia with a population of less than 1M people.
Based on the above, we were led to believe government would follow certain guidelines that included
economic development as a significant factor. We relied and acted upon this information, spending
significant time and resources, only to be eliminated before economic development was evaluated.
It was our understanding that the RFSQ process was being used to assess interest and to inform the
Province on potential solutions and companies in the market. Making a decision to limit the number of
vendors on a procurement of this magnitude, prior to assessing the full offerings, seems premature.
• The RFSQ document allowed for the evaluation team to seek clarification, and to reach out to
respondents as they saw fit. Even though the evaluation committee identifed areas that they could
have asked for more detail, or some clarification, they did not do so, and scored our submission only
based on the information provided. No questions or clarifications were sought, Section 3.2.3 & 3.2.4.
• RFSQ suggested page length for responses. This limited our ability to expand but Sections 3.2.3 and
3.2.4 gave the evaluation team a vehicle to verify, clarify, request supplemental information, as well as
request meetings with the consortium, we felt that staying closely aligned with the page lengths was
paramount to expanding the answers..
• Definition of the 5 point scoring system. What does each score mean? Procurement should have
provided these definitions so we can better interpret the result and ensure consistency. If they don’t
exist then there is no validity to the scoring.
• Conflict of Interest and Prohibitive Conduct Section (3.4 & 6). It appears as though our clinical
solution was discounted by NSHA, specifically two individuals, the CIO and the Director of Clinical
Applications for reasons never articulated to us. Both successful proponents, Cerner and Allscripts
had access to NSHA executives, key decision makers and were given the chance to present, to them
and in public and private forums. Requests for meetings by Evident and its consortium members were
all rebuffed.
• No one at NSHA has actually seen our clinical solutions in operation and thus to base such a large
transformational project on a 25 page qualification document, without evaluating or viewing the
solution or accepting additional information and clarification from us is highly irregular in any market.
• No references were contacted. Scoring of references provided, without actually calling them does not
allow for a verifiable score.
• No references were required for smaller hospitals. Evident consistently places #1 in the US in
hospitals <100 and <300 beds. According to Canadian Institution for Health Information (CIHI), only 2
hospitals in NS are larger than 300 beds.
• The RFSQ specifically allowed for all information, either in the process, or publicly available or known,
to be used in the evaluation. During the debrief we were told the opposite was done, and that they
did not allow for any information to be evaluated that was not in the proposal. Its been well
documented that Cerner is currently experiencing significant cost overruns, delays and poor
satisfaction in British Columbia. Doctors in BC are refusing to use the Cerner system, claiming it’s
negatively impacting patient care. The Cerner implementation in PEI was far from a success, as well as
the recent implementation at the Muskoka Hospital in Ontario. Much of the frustration stems from
poor implementations teams on the ground; resources are fully deployed outside of Canada.
• Last year Cerner won the US Department of Defense contract and more recently announced the US
Veterans Affair Department EHR project. Development and Implementation resources have been, and
likely will be directed at these multi billion dollar deals. The province of Nova Scotia is increasing its
cost and implementation risk by contracting to large multi-nationals with no skin in the game in Nova
Scotia.
• Non-binding pricing – Health Care Implementations in Canada have historically shown to run
significantly over cost, and to deliver under promise. To shortlist when there is no pricing guarantee,
no exploration of cost/time overruns or change requests is highly irregular. Evident was willing to fix
price many of the costs, taking the risk off the Province, and placing it on the Consortium. OPOR and
the implementation of a province wide CIS will be a long-term partnership. It would be important to
meet the people behind the solution and have meaningful discussions. No attempt by NSHA has been
made to evaluate the businesses behind Evident.
While waiting for what we thought would be a positive outcome of the RFSQ process (based on the
feedback from Government and NSBI), and in anticipation of the RFP phase of this procurement, the
Consortium continued to fully investigate the best deal for the province, we continued to invest by
augmenting our offering, and adding experts in their fields to evaluate and address any deficiencies. Our
expectation is much of the research and development would have been done in NS leveraging the local
companies, and Universities. This was always a key piece of our offering, but the RFSQ did now allow for
adequate measuring of this, not just from an economic development perspective, but also from a risk
mitigation perspective. Risk would be lowered by having a true local partner working closely with the
province, developing future road map enhancements, and building exportable Intellectual Property.
None of the Consortium members have seen a process like this, down selecting 2 vendors for the RFP
process that have only submitted non binding pricing, as well as not exercising their right to do a
functional assessment of the product and seek clarifications on a deal of this size.
1.4 REQUEST
• Ensure appropriate weighting be given to the economic development opportunity, research and
development, risk, and pricing certainty for Nova Scotia in any new procurement.
• Ensure any Conflicts of Interest be addressed and dealt with prior to moving forward
During the meeting Evident asked multiple times, “Why didn’t the Province ask for clarification”. There
were multiple responses given:
“The Province could not ask questions or request additional information to clarify answers”
“Information was not supplied in the depth they wanted it”
“They needed clarification….has risk that it may introduce new information (not permitted)
“No time in this process for clarification”
Clearly stated in the RFSQ document the Province had the latitude to verify, clarify and request
supplemental information. Neither Evident, nor any of our reference was ever contacted.
Within our Consortium we have never seen a process that doesn’t exercise the right to seek
clarifications on a deal of this magnitude. We believe this procurement has taken a very different path
than the 99% of transactional tenders that get evaluated. We believe that government has an obligation
to the taxpayers of Nova Scotia to be the most informed possible on the largest 1% of procurements,
especially in the instance of an RFSQ process with non-binding pricing.
Clarification or open dialogue on areas needing more detail would have positively impacted our score
and more than likely changed the outcome of the current process.
Definition of the 5 pts. scoring system was absent in the RFSQ document and at the debrief session.
What does each score mean? The procurement group should have shared these definitions with
proponents so we can better interpret the result and this in turn would also ensure consistency across
all bids. If this is not formally documented and can be provided to proponents then there is no way to
ensure validity to the scoring.
With respect to the individual Evaluation Criteria sections, the Evident Consortium has reviewed our
score and the comments shared at the debrief section and have identified numerous errors. For
example:
Comments
2.1 – “ONC Health 2015 Certified – core certs ok but 2015 not achieved (certified but noted that
measures are not demonstrated) / question on missing items, cannot ask for new information”
- (3/5)
The ONC Certification is public information, available at ONC CHPL website. Evident is
fully certified and should not have lost any points. No clarification requested.
2.2 – “Demonstrating logical data store for multiple legal entities - concern that there was not
enough evidence to support size and complexity” - (3/5)
2.3 – “Additional listed criteria and additional (8 core) abilities in 6 of 8 and a bunch of additional
ones” - (2/5)
Evident answer YES to all criteria. No understanding of the 5-point scoring system. No
clarification requested.
2.4 - Base content and rollout – “struggled with descriptions clarity / inconsistencies example 6
months in one section 4 years in another section” (1/5)
The question was answered without any inconsistences. No understanding of the point
system. Suggested length 1 page. No clarification requested.
2.1.2 Analytics:
Both references are much larger and complex than anything in NS.
3.2 - Expertise and technical clinical data and analytics – “workflow processes not enough detail
/ real-time data management, data customization not described / no extraction details
provided” (2/5)
Baylor College of Medicine, located in the Texas Medical Center in Houston, Texas, is a
health sciences university. It includes a medical school, Baylor College of Medicine; the
Graduate School of Biomedical Sciences; the School of Allied Health Sciences; and the
National School of Tropical Medicine. The school, located in the middle of the world’s
largest medical centre, is part owner of Baylor St. Luke’s Medical Center, part of the CHI
St. Luke’s Health system, and has hospital affiliations with: Harris Health System, Texas
Children’s Hospital, The University of Texas MD Anderson Cancer Center, Memorial
Hermann–The Institute for Rehabilitation and Research, Menninger Clinic, the Michael E.
DeBakey Veterans Affairs Medical Center and Children’s Hospital of San Antonio.
We push data standardization and not customization, which is what the province wants.
3.3 - Flexible reporting, - “real time reporting was not clear” (4/5)
No clarification request
3.4 - quality analysis and electronic reports and alerts (reporting and remediation) – “not
detailed enough” - (3/5)
No clarification request
3.5 - health service use attainment – “no demonstration of 3 of 5 programs (clinical program,
public health”,) – (2/5)
All HSU were answered even Clinical Program and Public Health. No understanding of
the point system. No clarification requested.
Pentaho’s health care customers have successfully addressed all five HSU categories
using its end-to-end data integration and analytics platform. The following provides
examples in addition to the above referenced use cases.
1. For direct patient care, health centres such as St. Antonius (referenced above) use
Pentaho for visibility into trends on patients’ vital statistics, frequency of clinical
visits, and treatment effectiveness to track quality and effectiveness of care
protocols and/or take preventative action.
2. For clinical program management, Pentaho helps organizations monitor trial
protocol activities and analyze trial results. Pentaho is also embedded by a partner
who is a leading SaaS provider of clinical trial software where it serves as the data
management back-end to provide an integrated view across all sources.
3. In health system planning and management, organizations like Loma Linda
University Health Care—an educational health-sciences institution with over 500,000
patients and 600 doctor—use Pentaho to align material replenishment with actual
usage, reduce staffing costs by matching schedules to patient census, and improve
the flow of patient discharges via real-time access to data.
4. For public health assessment, a Pentaho-based solution processes and analyzes over
100 million patient records across hospitals, skilled nursing facilities and home care
locations in 12 countries to create wellness scorecards that measure infection rates,
patient satisfaction and other outcomes against established benchmarks pertinent
to various specialties.
5. In research, Pentaho provides the data integration and analytics platform for cutting
edge initiatives such as the bioinformatics analytics projects that support human
genome sequencing lead by the top-tier medical school Loma Linda University Health
Care.
2.1.3 Technology:
“Strength - Economic development, benefits, local centre of excellence in NS, not outside
Weaknesses - Didn’t demonstrate understanding of OPOR size and complexity (didn’t describe risks and
how overcame them in other areas)
No evidence of similar scale implementations, phasing didn't take into account sun-setting McKesson,
risk in other implementations (demonstrating prior experience with enterprise implementations)’
Without knowledge of how the 5 point scoring system works, it is very difficult to
understand why we lost points in 6.3 – 6.5, we are confident in our answers. The Evident
consortium was also disadvantaged because we have been blocked out of the process by
NSHA. Both Cerner and Allscripts have had access to NSHA and have intimate
knowledge of what NSHA leadership vision would be for, “benefits and efficient gains in
NS”, as well as the future roadmap for NS.
2.1.5 Financial:
Score - (8.4/15)
7.1 - Audited statements – (3/5) -
7.2 - Approach to financing – (3/5)
7.3 - Describe plan for investments – (2/5)
Without knowledge of how the 5-point scoring system works, it is very difficult to
understand why we lost points in this section. We do not understand how the RFSQ can
ascertain our ability to raise capital. In 2015, Evident raised $250M to acquire our
largest competitor. We are financial very strong and Hitachi Data Systems (HDS) is our
partner. HDS is one of the largest companies in the world with the ability fund mult-
billion dollar projects.
In regards to Evident’s R&D investment. Evident does not capitalize software research
and development costs. Consequently, our R & D expenditures as a percentage of
revenues appear to be low. However this accounting statistic can be misleading when
used as the only measure of R & D activity. Where options are available in the
recognition of revenues and expenses, Evident opts for a more conservative approach.
The treatment of R & D software costs as an ordinary expense, as Evident has always
done, is universally considered a more conservative accounting approach. This being the
case, Evident has chosen to not expend the effort necessary to quantify the actual time
spent on R & D projects by the many individuals in our company involved in new
development outside our formal R & D departments, including almost every member of
our senior management team. If the time of all the people at Evident involved in the
development of new functionalities and products were accounted for as R & D expenses,
our R & D expenditures as a percentage of revenue would be significantly higher and we
are certain that it would be in line with industry averages.
2.1.6 Summary
Scores are often adjusted to achieve an outcome, as dialogue will always happen and the leadership will
influence people. Regardless of how independently people say things were done, everyone knows it’s a
reality and this is proven by the last point provided by NS Procurement during the debrief:
“Scoring was done independently and once all scores were done, the group met to discuss and
agree via consensus on final awarded scores. Averages were not used.”
Evident believes that leadership at NSHA have undermined the process and that a fair and transparent
process is not unfolding in NS. Cerner and Allscripts both had access to NSHA, which put them in a
favourable position to score higher in the RFSQ process.
Secondly, NS Procurement clearly did not follow their own process as outlined in this RFSQ. The ability
to verify, clarify, request supplemental information and meet with vendors where all options and none
exercised.
OPOR is neither a system, nor an IT project. It is a vision, strategy and culture for transforming
the delivery of care for Nova Scotians that is enabled by underlying IT systems including a core
clinical information system.
This is not a just technical solution to address a problem but more importantly the chance for true
partnership to better the province in anyway.
Evident understands the concerns raised by NS Procurement around references, tertiary and paediatric
care, size of the project etc. Technology is only piece of the puzzle, secondly it’s the people behind the
scenes, building and supporting the implementation. Aside from a world-class Consortium we have
assembled some of the best subject matter experts in Canada for this project:
Lydia Lee from KPMG is highly a regarded hospital CIO in Canada. She recently completed a 10-year
tenure at Canada’s largest tertiary care facility, the University Health Network (UHN).
Krista Laybolt based in Halifax is a Senior Manager at KPMG. Krista has extensive experience in NS
Health System and large enterprise HIS implementations. She successful ran the HIS Implementations at
Children’s Hospital of Eastern Ontario and a very large health system in Pennsylvania.
Tyson Roffey is highly regarded CIO with extensive experience in paediatric care. As CIO, Tyson oversaw
the HIS procurement and implementation at the Children’s Hospital of Eastern Ontario and was also the
CIO the Sick Kids hospital in Toronto.
Don Trotter, based in Halifax is one of the most senior and experienced solution architects in Canada.
Don has intimate knowledge of the healthcare ecosystem in Nova Scotia, as well as adapting the Evident
solution for the Nova Scotia requirement.
Evident and Consortium members have patiently and respectfully followed the process and will continue
to do so. All we are asking for is a fair and transparent process. We have a highly capable solution,
world-class Consortium, the best subject matter experts but no venue to tell our story in.
3 CONFLICT OF INTEREST
Excerpt from RFSQ
For the purposes of this section, the term “Conflict of Interest” includes, but is not limited to, any
situation or circumstance where:
(a) in relation to the RFSQ process, the respondent has an unfair advantage or engages in
conduct, directly or indirectly, that may give it an unfair advantage, including but not limited to
(i) having, or having access to, confidential information of the Province in the preparation of its
response that is not available to other respondents; (ii) communicating with any person with a
view to influencing preferred treatment in the RFSQ process (including but not limited to the
lobbying of decision makers involved in the RFSQ process); or (iii) engaging in conduct that
compromises, or could be seen to compromise, the integrity of the open and competitive RFSQ
process or render that process non-competitive or unfair; or
(b) in relation to the performance of its contractual obligations under a contract for the
Deliverables, the respondent’s other commitments, relationships or financial interests (i) could,
or could be seen to, exercise an improper influence over the objective, unbiased and impartial
exercise of its independent judgement, or (ii) could, or could be seen to, compromise, impair or
be incompatible with the effective performance of its contractual obligations.
“The RFSQ process was objective and fair with all respondents having equal consideration.”
“It was also stated that there were no potential conflicts identified with any of the bids.”
We are of the opinion that there is a significant conflict with Nova Scotia Health Authority and the 2
RFSQ qualified vendors, Cerner and Allscripts, in particular, NSHA CIO and Director of Clinical
Applications.
It was also made crystal clear in the debrief session that Allscripts has no business in the province of
Nova Scotia. The two NSHA individuals mentioned above have extensive knowledge of Allscripts but
have not provided Evident with any access and have acted disingenuously over the last 18 months, so as
to appear to have purposely blocked us out. To help further demonstrate our concern in this area we
provide below a timeline of interactions between Evident’s consortium and NSHA executives. Please
note that all information is publicly available to validate each interaction:
● February 17, 2016 - NSBI arranges 2 days of meetings with decision makers from ISD, DHW,
Academia, Health Minster and NSHA. Evident ‘s Chairman, the CEO and several senior VPs from
Mobile, Alabama attended. The CIO from NSHA accepted the meeting invite but was the only
person who did not show up for her meeting. She did attend a social event that evening put on
by Evident.
● February 22, 2016 - Evident emailed and hand delivered a thank you letter from their CEO to the
NSHA CIO thanking her for attending the event and requesting a follow-up meeting - no reply
received.
● March 21, 2016 - MOBIA VP emails CIO NSHA requesting follow up meeting - Response from CIO
“Although we are not formally in an RFP, we are not meeting with vendors at this time. I will
pass your name to the team, and they will be in touch once it is ready.”
● May 16, 2016 - Evident and MOBIA VPs run into NSHA CIO and Director of Clinical Applications
NSHA at Press Gang where she was having dinner with a vendor, Gevity Consulting. Gevity is
part of the Cerner Consortium.
(ISD and DHW CIOs inform us they are not aware of a lockdown on communications.)
● September 2, 2016 - The Evident Consortium submits an unsolicited proposal to the government
of Nova Scotia
● October 14, 2016 - email from the Chief Procurement Officer for NS informing that our
unsolicited proposal was rejected without any discussion or negotiation. In de-brief, we were
told we hit all the marks and encouraged us to participate in the formal procurement.
● October 2016 - CIO NSHA tells an Evident consultant that she is not impressed that ‘a small
hospital company’ thinks that they are able to service the needs of NSHA (large teaching
hospital). Also informs him that her Director of Clinical Applications comes from a hospital in
California where she implemented AllScripts previously.
● October 27, 2016 - NSHA hosted a public information session, ‘Let’s Talk Informatics - The Power
of AllScripts”. The presenter, an AllScripts VP & CMIO started the presentation by thanking
NSHA staff for the great evening at the Bicycle Thief and proceeded to promote the AllScripts
solution.
● October 27, 2017 - During the question period after the Allscripts presentation, a question was
asked, how will the province afford this system. NSHA answered with the financing model
Evident proposed in our unsolicited proposal. Our proposal was submitted in confidence.
● January 2017 - RFSQ was issued with a deadline of February 14, 2017 (there was a rectification
● April 27, 2017 - NSHA hosted another ‘Let's Talk Informatics’ public session highlighting another
vendor’s product...Cerner’s lab information system.
● June 7, 2017 - Procurement website lists Cerner and AllScripts as the 2 successful vendors.
● August 2, 2017 - Screen Shot from CIO NSHA linkedin page, 2 note the company she is following
on the right side:
● August 2, 2017 – Screen Shot taken of Director of Clinical Application NSHA Linkedin page, note
8 out of the 9 accomplisments are for Allscripts products.
4 CONCLUSION
Evident and our consortium members believe that the OPOR procurement members we have interacted
with to-date have acted in good faith. We are disappointed that the RFSQ process chosen for this
important initiative could have been used more effectively to get the best deal for the Province of Nova
Scotia.
We believe the RFSQ provides you with many options to move forward and we would like to thank you in
advance for reviewing the concerns and considerations raised in this complaint.