Complexity of Possibly-gapped Histogram and Analysis of Histogram (ANOHT)

Abstract: Without unrealistic continuity and smoothness assumptions on a distributional density of one dimensional dataset, constructing an authentic possibly-gapped histogram becomes rather complex. The candidate ensemble is described via a two-layer Ising model, and its size is shown to grow exponentially. This exponential complexity makes any exhaustive search in-feasible and all boundary parameters local. For data compression via Uniformity, the decoding error criterion is nearly independent of sample size. These characteristics nullify statistical model selection techniques, such as Minimum Description Length (MDL). Nonetheless practical and nearly optimal solutions are algorithmically computable. A data-driven algorithm is devised to construct such histograms along the branching hierarchy of a Hierarchical Clustering tree. Such resultant histograms naturally manifest data's physical information contents: deterministic structures of bin-boundaries coupled with stochastic structures of Uniformity within each bin. Without enforcing unrealistic Normality and constant variance assumptions, an application of possibly-gapped histogram is devised, called analysis of Histogram (ANOHT), to replace Analysis of Variance (ANOVA). Its potential applications are foreseen in digital re-normalization schemes and associative pattern extraction among features of heterogeneous data types. Thus constructing possibly-gapped histograms becomes a prerequisite for knowledge discovery, via exploratory data analysis and unsupervised Machine Learning.