1	Results of IAC Study of Metrics in Electronic Records Management (ERM) Systems
2	Introduction and Principal Conclusions How does one measure the impact of an ERM system to the bottom line business or mission of an organization? What is the business case for an enterprise ERM system? Principal conclusions: No silver bullet No universal COTS tool or product No one metric captures the success of an ERM system and relates unambiguously to the bottom line Notwithstanding: Some common categories of metrics in use today Some metrics less burdensome to capture than others Some metrics just reflect a measure of IT system performance Some metrics reflect mission success more directly than others Measurement of ERM performance is currently immature Most measurements tend to be IT-related rather than related to records management itself Valid comparisons of ERM practices across organizations are difficult to make, and probably should not be made
3	Bottom Line The inescapable conclusion: There is no simple, single answer! There is no Swiss Army Knife-like tool Tradeoffs must be made to arrive at metrics that are: Meaningful to measure ERM success (e.g., “good” vs. “bad” metrics), and Not too burdensome to capture on an enterprise-wide basis “What gets measured is what gets done” Aggregation of metrics into a single coherent picture of bottom line performance is problematic
4	Concerns to Consider Metrics for Public Services Relating to ERM Spirit of the eGovernment initiative is to provide a Government that “works better and costs less.” Quantifiable and well-defined ERM metrics relating to capacity, throughput, security (especially data and records integrity), assured service availability, ubiquitous access, lower cost, improved turnaround times, etc. are of interest. Also concerned about particular metrics that are unreliable, non-specific, intractable to interpret, or too burdensome or onerous to collect.
5	Major Factors to Consider Who is the Consumer? Nature of the “consumer” is an important factor “Who” and/or “what” the metrics are sampling “Public at large” Specific customers Agency/company employees Federal agencies, Other government agencies corporations, or Foreign users, etc. What is the ERM Business Practice? What specific “bottom-line” agency and/or industry business practices the metrics supported. For example: Servicing FOIA requests Support for legal discovery Historical research Genealogy Auditing and controls Regulatory compliance Public information dissemination Statistical analysis Archival records management Grants management ERM systems operations and management Specific mission support (e.g., medical, environmental, emergency and disaster, defense)
6	Principals in Defining ERM Metrics Not everything that can be measured needs to be measured nor should it be Metrics should have a purpose for continuing improvement Best to design the capture and management of metrics into a system upfront or provide for an SLM approach Important “paper vs. electronic” paradigm issues to be understood
7	Broad Categories of ERM Metrics Access to ERM Services Accuracy Capacity Efficiency Participation Productivity Search and Retrieval System User Satisfaction Utilization Legal ^*
8	“Good” vs. “Bad” Metrics Many metrics are potentially ambiguous, intractable, unreliable, or burdensome to capture Among the more problematic metrics: Record search time Record retrieval time Number of seats (or licenses) Session time, and the Raw number of records in the system All of the above can be captured However, interpretation of each can be quite controversial A long session time, for example, could be indicative of great success or utter failure Search times can be curiosity-driven as in surfing the Web Level of commitment and persistence of user can not be easily measured Some people are just better than others at “finding things” Training, domain knowledge, and time-of-day can be important mitigating factors
9	Sample Candidate Metrics for ERM Systems
10	Sample Candidate Metrics for ERM Systems (cont.)
11	Sample Candidate Metrics for ERM Systems (cont.)
12	Summary