Design principles for big data management

I recently faced the challenge of designing a big data management interface. Having not worked on a big data application before, some of the fundamental design patterns I’m used to using were challenged.

One such UI pattern is that of create, update, view (single), delete and list. This pattern is one of the basic building blocks when managing data. However, such a granular approach doesn’t fit in a big data environment. As the scale of data increases so does the cost of human interaction and simultaneously the usefulness of individual data points decreases. Instead, I found new patterns emerging. The following are some of those patterns.

• Automate everything. If something can’t be fully automated then provide the known details and let the user finish the task or make a change and ask the user for approval.
• Allow actions to be applied across the data set, never just to a single item.
• Aggregate and report on data. Raw data is (almost) never useful.
• Search should be the primary method for retrieving a single item.
• Display a list of data only when you can provide a small enough subset that it can be read and acted upon.

I think there are many more challenges and principles in big data design, but I’ve found these rules to be a good starting point.