To all of my engineer friends in the med tech sector. Take note, this is one of the most elegantly simple and functional products ever developed. (IMHO).  Aim for simplicity. 

It's taken fewer number of years for the cell phone to become as ubiquitous as e-mail.

Until, e-mail reached the point of the reliability of a dial-tone - not to many years back -  it could not have been deemed ubiquitous. There are many factors, the snow ball effect, that needs to come to fruition simultaneously for any single technology to reach the point of being truly ubiquitous.

In the case of email you have storage - the cost of storage, and storage technological advancements as major components that needed to mature in order for email to be part of the dial-tone reliability age.

Also, data-center advancements in cooling, smaller form factors, basically servers doing a lot more, in less space with less power and networking from the cost and technology advancements of routers and switches - and emergence of widely used protocols to emerging standards that made it easier for everyone to talk the same language without the overhead and errors of bridges and translation.

Note, that it wasn't to long ago when servers spoke IPX/SPX, DEC LAT, NetBeui we were working with Token Ring, Banyan Vines, etc. It's rare to find any of these today on the average corporations LAN.

All of these together, have combined to bring the reliability of a dial-tone to e-mail.  And it didn't take long.

I find it amazing, how so much attention - from a compliance, records management, classification, long term retention, legal discovery, etc. perspective - has been paid to electronic communications over the last 5 years, yet another technology that has evolved to the state of being ubiquitous, is used more so and at much lower costs than email, and is as creating a new wave of text messaging, is as prevalent a communication tool, if not more so, than email has received so little attention - the cell phone.

A quick look at the industry which has, in essence, started it all is the financial services industry, specially the broker dealer space. They gave birth to the e-mail supervision industry with regulations which required that certain e-mails be supervised. The regulations were broad enough that it gave room for wide interpretations and with little to no teeth , at least early on, enforcement and compliance was lackadaisical.

Vendors came and went and emails were supervised, but not retained, since the regulations didn't require retention, just supervision.

This went on for sometime, then post ENRON, a definite point in time, came the requirement to 'retain' these same communications, plus the inclusion of instant messaging with email.

These were definitely a monkey wrench, because the same system that was designed to supervise email had no concept of long-term retention and although instant messaging looked like another message on the surface, the issues took a while to iron out, and generated yet another breed of vendor specialized and focused on instant messaging capture.

It all amounted to higher volumes, increased processing and a new technological direction was needed to meet the new requirements, archival software.

Today, all of us - from IT personnel to business and legal people -  find ourselves engaged in conversation where data storage is the focus.  Whether we are talking about buying a laptop for personal usage, "how much disk space do we need", or involved in a business and planning conversation at work - storage jargon is more and more becoming common speak today.

Terms such as megabytes, gigabytes and terabytes are no longer reserved for the elite few who specialize in managing the company's storage infrastructure. Today, everyone is familiar with those common terms and we use them as part of our every day vocabulary. In fact, its probably safe to say that we all refer to megabytes, gigabytes and most recently we now talk of  terabytes, in the same way that we talked about kilobytes and megabytes a decade or so ago.

I can distinctly remember buying a 10MB hard-card that that slipped into a PC motherboard slot, taking up the space of two slots and running very hot, and the  cost was around $800.USD (I think).  I needed the additional space for an application database that I had built to run on an Intel 80286 PC, and I had to research the best use of company funds for adding the additional space to a single PC.  I settled on a device called a hard-card, it used the ISA slot on the PC, it needed a software device driver to be recognized by the operating system and the heat from the device nearly melted all of the other parts of the PC, but a small investment in a cooling device, (a fan) underneath a desk somehow managed to keep everything cool enough to continue operating without any interruption. I think the internal disk on the machine was probably 10MB. The time-frame was mid-80's.

A short-twenty year's later, we talk about gigabytes and even terabytes with much less significance (and costs) than that 10MB of disk space back in 1987.

Now, given that the average business person, who doesn't live in the IT centric storage field, probably doesn't have an appreciation for the differences, between all of these terms mega, giga, and terabytes - I thought it would be an interesting exercise to use a simple format that we can all understand "money" as an analogy to see some of the differences for what we talk about in every day storage speak.

So here it goes- at the most basic primer level, when we talk about mega-bytes, giga-bytes and tera-bytes, the lowest common denominator (although a bit) we generally talk about is a byte of information. So [mega] is a million bytes of information [giga] a billion (makes you wonder why giga wasn't called biga-bytes) and tera is a trillion and so on, it goes higher, but we are probably a few [short] years away before we talk about "peta" and "exa" in the way that we talk about "giga" and "tera" today.

So in the same way we compare a million dollars to a billion dollars, this is essentially the same example, however it should remind us of the extra-ordinary differences in the amount of storage that we are generally talking about in our day to day business conversations.

Let's say your rent or mortgage payment is $4,000 dollars a month - here's the difference between a megabyte, gigabyte and terabyte. A megabyte will pay your mortgage payment, month after month, for the next 21 years. On the other hand a gigabyte, will keep your payments going without interruption for the 20,833 years. That's the order of magnitude of difference, 20 years vs. 20 thousand years. Now, think about how often we use the term "terabyte" in our conversations - a terabyte will keep your mortgage payment clear and hassle free for the next 20 million years! Now think about the following, organizations today, are archiving for time and all eternity (this means no clearly defined retention period, where the data will be removed from the storage devices on a regular basis) electronic data in the order of one terabyte per day.

The exasperating problems that have led to the storage overload can be summarized in one word, "e-mail".  This overload and casual approach to extra-ordinary amounts of data, and requirements for the data stored, such as the ability to search, search fast and accurately, extract the data, keep it in it's original form, retrieve it in its original form, and the list goes on cannot be taken lightly. It is a monumental problem from a number of angles and across a number of disciplines where the storage component is just one of many, yet we tend to take a very simplistic approach to solving the issue of archival - buy some software, get some storage and were done.

So, if you find yourself taking about gigabytes and terabytes, remember that just a few years back we were talking kilobytes and megabytes, so if your strategy is along the five to ten year planning horizon, you will be better to start contemplating how you will tacke the problem of petabytes and exabytes, its coming sooner than you might think.

For a different perspective visit the MegaPenny Project.

Depending on the industry and the typical damage periods (i.e. Anti-Trust where the damage period can go back 10-15 years on averge) most of the data needed for responding to legal discovery will reside on tape.  So if you’ve recently implemented an archival strategy for capturing e-mails, where you can use the system for legal discovery, you may still need to deal with tape restores for quite some time, years even.

It’s prudent to consider tape restoration to the archive as part of your implementation and strategy upfront.

About the picture. PowderHorn 9310 tape library

The cost for tape restoration can usually be high and typically involve third parties other than your archival vendor to deal with the factory style logistics needed for managing 100’s or maybe 1000’s of tapes.  The key need however resides within the archive tier and it’s ability at the component layer to handle lower level e-mail formats such as internet standard RFC822, EML, DXML, etc. basically as many formats as possible, so that the avenues for re-ingestion are flexible.  What’s more important however is the ability to separate or mark the data that is coming from tape, as such. Perhaps, keeping more than just a virtual store makes the most sense.

One of the key pieces of information that you lose when restoring from tape is access to unwind the address books or directory servers containing group based addressing information. So the search for all e-mails to [Fred Smith] will only yield those messages where Fred Smith is listed explicitly in the To: field. All of the messages where Fred Smith received messages as a result of being a member of a group will not be presented in the search.

Some vendors enumerate the information backwards by simply identifying all of the mailboxes that contained a certain message and then deducing that if a message was in your INBOX and you are not on the address line, then you were a member of the group distribution. That doesn't catch the use case where Fred received a message, opened it, read it, and then deleted it, before the next backup cycle - if the email backup system is backing up mailboxes then the tape restoration, even with a deduced distribution list is completely unaware of this transaction.

In the case where the group lists are preserved, the system would know the more pertinent piece of data, which is that Fred received a message, but its not in his Inbox.

This is one reason why it’s  a good practice not to mix newly captured e-mails where the distributions lists are expanded in real-time, with legacy data imports where access to the distribution list which was active and current at the time was integrated separately - the context for errors, questions and further research really do require separate approaches to real-time vs. legacy email data.

Lastly, once the tape data is successfully ingested into the archive, and now managed via policies which govern it’s expiration based on categories, destroy the tapes, for legacy data they no longer serve a purpose and represent potential risk!

Instant Messaging Summary:

A criminal defendant convicted of assault claimed that instant messages threatening the victim should not have been admitted without authentication evidence of their source from the Internet Service Provider or the testimony of a computer forensics expert. The court rejected this argument, which it characterized as having the court create a whole new body of law just to deal with e-mails or instant messages.

Although the court recognized that such digital messages are inherently unreliable because of their relative anonymity and the fact that while an electronic message can be traced to a particular computer it can rarely be connected to a specific author with any certainty, the court found that the same uncertainties exist with traditional written documents.

Thus the court saw no justification for constructing unique rules of admissibility of electronic communications such as instant messages. Therefore, the court held that under Pennsylvania Rule of Evidence 901 circumstantial evidence, such as the contents of the writing and surrounding events, is sufficient to establish authenticity.  In this case, the circumstantial evidence establishing authenticity included the fact that defendant had acknowledged his first name in one of the instant messages and had failed to dispute having sent the instant messages in verbal discussions shortly after their occurrence.

E-Mail archival has taking several twists and turns over the last few years.

The primary one is the use of the archive for legal discovery, sometimes referred to as eDiscovery.

This has left many vendors flat footed and the reasons are in fact quite simple.

Number one, the same product that you, if your a commerical software company,contruct for archiving and stubbing / the process of removing the e-mails from the mail server store, and relocating the physical bits to an archive server, and creating a link so that the end-user can seamlessly retrieve the bits from the archive server instead of the e-mail server.

That solution value proposition is simple and strong.

You can reduce the storage on the primary mail server, which unto itself has many benefits, to many to list, but now your server and server administrators can focus on the main task of a mail server - to "process" e-mail - and not spend expensive cycles on managing storage and all of the sundry IT issues that account for storage management, which are voluminous, complex and expensive.

So, the point, is that the construction of a product to solve this problem is not the same product construction plan that you would necessarily use to solve another distinct problem called eDiscovery. These two are at odds with each other.

Vendors who were focused on pure play e-mail archival are now left flat-footed and looking for third party hooks to help shore up their product lines.

Basic rule, is that re-architecting commerical platforms is difficult, and once you have customers in production it’s literally open heart surgery, so you as the customer now, working with a pure play e-mail archival solution, is really looking at 2 or 3 vendors coming together to solve your business problem.

Both you and the vendor will be constantly challenged with EAI (enterprise application integration) and the final cost of a working solution can be 10x your original license price when it’s all said and done.

Bottom line, is that eDiscovery, legal discovery, compliance elements, storage consequences, cannot be after-thoughts to an e-mail archival due diligence.

The consequences for not considering the bigger picture upfront are costly from a capital expenditure perspective and more importantly the risk to the business for systems which are now error prone due to the inherent disconnects between the varying technologies.

When we are going down the road of e-mail archiving for regulatory reasons is there a thing as 100% complete, or is 5 - 9’s accuracy good enough?

What is the consequence for correct and complete as it relates to data archival systems deployed for compliance, legal discovery and ultimately corporate risk management.

If you are challenged with the requirement for capturing all electronic communications (inbound/outbound) what mechanism, process or IT controls can help you to achieve 100% accuracy or better put - can guarantee both complete and correct results? Even in the best case scenario can you ever really account for technology outages, software bugs (that can occur at every layer below the application, firmware for example on a hardware device), or even good old fashion human error. Although the vision of an archival system gives you the impression that you are dealing with a very static durable almost simple piece of technology that houses long-term data - nothing could be further from the truth.

The reality is that archival systems technology has evolved more over the last two years, than the entire previous decade - which included the internet, TCP/IP, local and wide area networking and more? Hard to imagine, but true.

This has been due to the regulatory shift that has occurred over the last few years, as a result of e-mail being required for long term retention (archival) in the financial services sector - all due to corporate malfesance. The technology shift and subsequent consequence for archival systems has been in having to extend the arms of the archive and reaching out to actively participate in the details of - how - the data is captured from it’s source systems. Traditionally, archives relied on the source system placing content some place where it could be picked up and then archived.

{A lot like dropping a laundry bag at the back door and having someone from the cleaners come by and pick it up}

Well imagine if the laundry pick up service didn’t have the bag of laundry waiting for them and instead they had to open the front door to the house and every house in the neighborhood, and deal with every type of security system in order to enter. The comparison is the archival system knocking on the door of many different systems (i.e. MS Exchange 2000, 2003, Lotus Notes 5,6,6.5, IM (instant messaging systems) ERP, and integrating into each distinct security system from the application to the OS, plus centralized directory sources. Now, can you imagine the laundry pickup service person walking through your home and locating all of the dirty laundry, opening up every draw, closet, and so on… Then drawing conclusions on each piece of laundry; which need light starch, heavy starch and so on…

The picture is pretty insane, but that is essentially the extended role of the todays archival technologies and more importantly - its required - if they are to remain both viable and competitive in solving problems for businesses - especially in this climate of heightened awareness of corporate content, security, audit requirements and the overall pertinence to protecting an organization reputation and profit.

With all of the moving parts involved in todays processes for archival technologies its critical that these parts are all part of a single comprehensive system - the alternative is an EAI approach and dealing with connecting these moving parts across different technology vendors. This is a task that even in the best case scenarios leaves major exposure with respect to both completeness and correctness of the overall system. If we evaluate completeness and correctness and 100% through several nines of reliability/accuracy what are we really talking about?

Well at a very basic level if your organization message traffic is 5mil messages/day completeness would account for capturing and archiving each of the 5mil messages transmitted each day. At the end of 5 years your archive would contain 5mil*264 (business days)*5 =6,600,000,000 (over 6.5 billion records).

If your ability to deliver completeness ranked at 99.9% reliability your 5 year archive, used and relied on for compliance and legal discovery, would be missing over 6.5 million records. 

{Can you afford to knowingly miss access to 6.5million records?}

This is the overall importance of completeness. An EAI approach is not the answer for delivering complete record archival, the disconnect between systems present risk for errors and inability to manage processes end-to-end; a single flexible platform gives you all of the tools and redundancies needed to achieve to 100% completeness.

A modular approach through a single platform to the archival process is paramount, as logging can be done at each module which provides the ability to systematically reconcile and rollback in the event of process exemptions. Correctness is yet another area of major consequence that is often ovelooked - and it has to be combined with completeness in order to deliver reliable data archival. Data has to be more than just successfully captured, indexed and archived, it must be processed correctly.

{Indexing errors have consequences on correctness, while you have the "complete" record it may not find it's way into a result set if the index was not created in it's entirety}

A simple example can be made with record categorization.   If a rule is applied incorrectly at the program; level, then data can be lost down-stream, accidentally deleted, not found or even not provided in the case of legal discovery (when it should have), etc. A more complex example can be an indexing operation, the details of indexing are complex, what happens if the indexing operation of an attachment containing 100 pages of MS Word fails to index pages 99-100?

The record, while complete is not 100% correct. The consequences for correctness is high.

The audit logging of events such as capture, indexing, combining, categorizing and nearly every discreet process involved in the overall archival operations must be capable of system logging.

And as important, if not more so, these log files must be monitored and automated to deliver alerts and triggered events so systems personnel can be alerted to and participate in re-repocessing information, while maintaining the full integrity of the data and the correction processes, when errors in key processes occur.

Few vendors have grown their systems organically and are delivering the required capabilities end-to-end through a single system approach this is the better and less error prone approach needed in todays business climate for both technical and business reasons - and most importantly combining both correct and complete.

Backup technologies, as they relate to the actual hardware and software involved in the backup process, have and continue to improve - more density, faster robotics, etc. etc. - it's an extensive list. However, what has not changed much over the last year is the actual "process" of backup execution.

The process hasn’t changed much in over a decade, it’s a well tuned systematic process of incrementals, fulls, swapping tapes, sending tapes off-site, etc. As prices for the backup technology have dropped overall, their has been a lack of attention paid to the actual cost of data stored on tape, but the cost today, is being fueled by a completely different rationale - compliance.

{For the purpose of this discussion let's define compliance simply as governance of data as it relates to an organizations legal and regulatory liabilities or requirements}

It’s no longer prudent to keep data that may be subject to discovery in a billion dollar class action lawsuit locked away, safe secure, and accessible on a tape - just waiting to be "discovered" by lawyers who know how to ask for the information.

So, what does that mean to today's backup approaches?

I don't know a single systems administrator that will put his job at risk by not backing up their e-mail stores, just one example of content.

But what's the risk to keeping millions of e-mails lying around on backup tapes?

Well Morgan Stanley was just fined 1.4 billion dollars. That should be reason enough to call together all the departmental CIO's and business unit heads and talk about how the organization is executing backup today.

I have some concrete ideas and practices which involve the use of archival technologies in relation to backup for securing and protecting data in a manner that delivers more protection with respect to risk.

It’s an interesting dilemma that requires a fresh look at process, existing and new technologies, and how they can be re-deployed to meet both the old and new challenges facing today’s corporations.

Every IT manager should evaluate how and what data is being written to tape, how it is being stored, where, how many copies, and what is the destruction policy in relation to legal or regulatory requirements and or corporate policies which are well documented.

To delete or not to delete, this now appears to be one of the holy grails of the e-mail archival industry, and it’s extending beyond regulated industries. On the surface, you would think that the regulatory industry is the best suited for deleting e-mails as the regulations are very specific for how long particular e-mails have to be kept. What more could you ask for, you have it in black in white, from the government in a loud booming voice "thou shall keep the Equity traders e-mails for a period of 5 years, thou shall only delete it after the prescribed period has come and gone".

So what’s the issue with deleting that e-mail on it’s 5 year anniversary date? With today’s systems you can schedule the deletion for the exact second it reaches it’s 5 year anniversary.   So again, what’s the big deal, well let’s take a look at look at it from the opposite perspective. 

What happens when you don’t delete the record. The answer to that one can be found in the Federal Rules, the pertinent rules are 26 and 34, which regulate the production of evidence. These rules make electronic information available for broad discovery, with some protections for the party whose information is being sought, these rules are currently under review with proposed amendments before congress, however let’s see how they can effect whether you delete or not today. Rule 26, states that all parties in litigation must disclose "a copy of, or description by category and location of, all documents, data compilations, and tangible things in possession, custody, or control of the party that are relevant to disputed facts alleged with particularity in the pleadings" FED. RUL. CIV. PROC. 26(a)(1)(B). Rule 26 also provides the scope of discovery: "Parties may obtain discovery regarding any matter, not privileged, which is relevant to the subject matter involved in the pending action . . . including the existence, description, nature, custody, condition, and location of any books, documents, or other tangible things." FED RUL. 26(b) So, if you have it, and it’s relevant and not privileged you are compelled to turn it over.  The protections under Rule 26 also defines discovery limits: "The frequency or extent of use of the discovery methods . . . shall be limited by the court if it determines that: (i) the discovery sought is unreasonably cumulative or duplicative, or is obtainable from some other source that is more convenient, less burdensome, or less expensive . . . (iii) the burden or expense of the proposed discovery outweighs its likely benefit . . ." FED. RUL. CIV. PROC. 26(b)(2). In addition, Rule 26 allows a court to authorize a protective order to protect a party from "annoyance, embarrassment, oppression, or undue burden or expense" FED. RUL. CIV. PROC. 26(c).

These protections essentially are designed to bring some law and order to the process which is admittedly out of control with respect to both electronic evidence and even paper records. The rules simply were not written to account for today’s massive volumes of both electronic and paper records.

So, back to the small matter of deleting records. Why not delete them if you can?

Otherwise you will have to provide them, if they are relevant to a particular legal matter. If you are being litigated for the largest deal your company ever executed and the government is requesting every piece of electronic information from or to these three senior securities traders in your organization, would  you want to turn over less or more information. The answer to that one is obvious. So why keep the records that you are allowed, by the same laws that you are trying to comply with, to delete.

The answer is simple, fear.

Fear of doing the wrong thing or more relevant lack of confidence in the electronic systems, people, processes and internal policies that are standing behind the delete key. And here is a dilemma where spending millions of dollars on a top tier consultancy to devise the processes, pick the systems, train the people, and possibly even hit the delete key doesn't make a difference. For good or bad, Enron and Sarbox have changed the world’s business landscape, so much so that it is spreading from the US to other parts of the world. The consequences for doing the wrong thing will be your problem and burden to bear, and Section 802 of the Sarbanes Oxley act talks to those penalties. There will be no one to point the finger at except yourself.

So, is the answer to keep everything? It’s certainly a strategy, however not a very good one given the risk. The answer is to take full and absolute control of your electronic systems, people and processes. Build internal confidence, training, re-training, certifications so that everyone is aware of the processes that govern your electronic data. Start with the most critical first, e-mail. Have a plan for what’s second and third on the list, and execute according to a well documented plan, that is part of your official corporate archive. And, lastly a small fact that seems to go unspoken, but execute swiftly and publicly for corporate violations of compliance matters. {My kids high-school hand-book is a good example, strike another student and receive an automatic 10 day suspension, no questions asked. And yes, I have been in the principals office pleading my kids case "the other kid pushed him first", no level of articulation was going to make a difference, the punishment was executed and swiftly}.

Within a system a record targeted as compliant should follow a specific chain of custody which can be re-produced through verifiable audit logs that are designed and sequenced to electronically prove, to the satisfaction of forensics experts, that what the audit logs reports is what actually occurred. A file plan for example, which dictates how a record from the Equity Trader is managed through it’s five year life-cycle should be recorded as part of the corporate archive, and if the file plan changes, the new file plan should be recorded and the electronic audit logs should be updated, and the states of times copies of the audit logs should be stored serially by date of occurrence within a secure tamper proof system.

These small level’s of detail within large systems with many moving parts will pay dividends in proving that even wrong decisions in data deletion were done without malicious intent, and under the umbrella of good faith, because you have designed full transparency into your entire governance platform.

What is needed within your overall systems is the element of “transparency”, shareholders tend to like that, and so do the regulators as well as the courts.

Proving what you did, how you did it, fully, is as important as to why.

You shouldn't keep records longer than needed, unless you make a conscious corporate decision based on the “smoking gun” factor. And that is that it is just as prudent if not more so, to know what the other side may possibly already know. Just because you deleted the Equity Trader’s mail after five years, doesn't mean that some other party joined in the same litigation didn't delete their corresponding copy. So one of your records may be turned over as part of another party’s discovery request, and you are potentially unaware. That is a viable reason, if made consciously to keep records, but certainly not because of fear.

It should also be noted that the current proposed amendments to the Federal rules discussed here, take into account that, based on the massive volumes of electronically stored information involved in particular legal matters, that errors in the discovery process are likely to occur, and they will not penalize one side or the other for accidentally turning over records that are not relevant or privileged. Basically, you may be able to call time-out and ask for a "do over". ESI or Electronically Stored Information will soon be recognized more fully at the Federal level, this will certainly make the years to come more interesting and challenging for providers of systems which manage ESI for the purposes of compliance and legal discovery.

Let’s suppose that a user’s local hard-drive (or shared file system) contains the following directories with the file entries listed below:

>>directory1>projects>ibm>file1.doc (note, these are the same file)
>>directory1>projects>ibm>file2.doc
>>directory1>projects>ibm>file3.doc

>>directory2>projects>oracle>file1.doc (note, these are the same file)
>>directory2>projects>oracle>file4.doc
>>directory2>projects>oracle>file5.doc

Now, your company is joined in a legal suit and received a discovery request for relevant documents to the legal matter. You are now on the receiving end of having to find and produce relevant documents in a very important legal matter. On the surface the request seems clear, "all relevant documents", well after many rounds of legal back and forth (the discovery order is to broad, etc.) the discovery request is now understood and amended to specify "documents" containing the word "IBM"

Now comes the conundrum.

Let’s cover the simple scenarios first, if file2 or file3 contain the word "IBM" they are clearly discoverable pursuant to the request and should be turned over.

But, what if file1 does not contain the word "IBM", though it is clearly in a folder named "IBM" should this document be produced?

If the supplier of the information withholds the document pursuant to the agreement that only "documents" containing the word "IBM" are relevant, is the supplier within his rights to withhold the document?

Let’s suppose that this document is withheld, but later discovered under a broader discovery request, will the opposing counsel claim foul and will a judge potentially find that while the letter of the discovery request was met that the company’s action was in bad faith and although the "document" did not contain the word "IBM" that it was obviously relevant, having been stored in a folder named "IBM" and should have been included - and the courts has no choice but to find that it was excluded intentionally and impose sanctions or even worse instruct the jury that there was tricky business afoot by the company and this should be considered as they deliberately withheld this document, (ouch).

What about the fact that the same document was in a folder clearly marked "Oracle" wouldn’t this be enough to prove that this document was relevant to "Oracle" and not IBM ? By the way, the contents of the document did contain the word "Oracle" and not "IBM".

Ah, the never ending complexities of discovery, a lawyer with an engineering background in systems can probably have a field day.

And, the point is exactly that - these are all issues, questions, suppositions, assumptions for legal counsel, many with answers that rely on their ability to make (or not) successful arguments in written motions and arguments in front of a judge -  as well as digging up any relevant case law and or legal precedents to support their opinions..

These are not questions suitable for software vendors providing the underlying technologies that support responding to legal discovery requests.

However, there are some very pertinent questions that the above scenario(s) do present for software vendors that are more important to your companies ability to support any number of scenarios; I’ve addressed the importance of flexibility before and here we go again; a system of record, namely a durable active archival system must support documents, e-mails, instant messages, other electronic data in the form of XML messages (as an example) that represent transactions in various states usually traversing through a series of queues within a message bus, end of day or intra-day report out-put such as FX tickets, trade-confirms and end of day confirms with cancel corrects; all of this data must be recorded as states of time, categorized, and marked for expiration based on regulatory and internal policy.

A vendors ability to handle the sheer volume associated with the above is critical.

Now, if a record must be maintained for 7 years, an organization may take the approach of expiring the document on the 7 year anniversary to the second, or go 7 + 1 year, etc.

Systems should provide event based flexibility to support liability and risk factors, for example the logic and reasoning that goes into expiring a tax document after 7 years is pretty simple, that’s what the law allows.

But, are complex business processes for global enterprises ever that simple cut and dry, well not in my experience.

Expiration logic which can be expressed as *expire* tax documents in 7 years, if account numbers beginning with "Z" ("Z" equals international) have passed all external audits (maintained in another system) and if company names contained within the documents (as a indexed field) are not part of any legal matter irregardless if the tax documents are on legal hold, also based on the previous triggers send these documents to the legal department for review and purge or retain with a recorded attestation.

This example is probably more in keeping with complex business processes;

So, back to the matter of :where: obviously where documents originate from, reside, or are moved to can have consequences to their admissibility.

So vendors should record this information as part of a "complete" record, referring back to another post, we have discussed the differences between complete and correct, they are distinct and both critical.

How a vendor tracks where and to what extent, such as the physical asset is important, see that your vendor understands and incorporates "where" as part of their overall systems.