TeAM YYePG Digitally signed by TeAM YYePG DN: cn=TeAM YYePG, c=US, o=TeAM YYePG, ou=TeAM YYePG,
[email protected] Reason: I attest to the accuracy and integrity of this document Date: 2005.07.10 23:04:19 +08'00'
Pattern Recognition Letters 26 (2005) 1632–1640 www.elsevier.com/locate/patrec
On the compact computational domain of fuzzy-rough sets Rajen B. Bhatt *, M. Gopal
Control Group, Department of Electrical Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110 016, India Received 23 June 2004; received in revised form 13 November 2004 Available online 24 February 2005 Communicated by W. Pedrycz
Abstract Based on some properties of fuzzy t-norm and t-conorm operators, the concept of fuzzy-rough sets on compact computational domain has been put forward. Various mathematical properties of this new definition of fuzzy-rough sets are discussed from pattern classification viewpoint. It is established that the proposed approach identifies various patterns in the sense of fuzzy-roughness, in addition to providing deeper insight into various concepts of fuzzy-rough sets. 2005 Elsevier B.V. All rights reserved. Keywords: Compact computational domain; Fuzzy-rough sets; Pattern recognition
1. Introduction Rough set theory provides systematic approach to knowledge discovery from inexact, noisy, or incomplete information (Pawlak, 1991, 1999, 2002a,b). The central concept in rough set theory is to approximate a target set through crisp partitions generated by equivalence relation, by a pair of exact sets called the lower and the upper approximations. Roughness emerges when there is a one-to-many mapping from equivalence classes
*
Corresponding author. Tel.: +91 11 26596133. E-mail addresses:
[email protected] (R.B. Bhatt),
[email protected] (M. Gopal).
to target sets. Several applications of rough sets in data classification, data analysis, machine learning, and knowledge discovery are of substantial importance (Slowinski, 1992; Pal and Skowron, 1999). Since, rough set theory relies upon crisp partitioning of the dataset, not considering the degree of belongingness of real-valued data, important information may be lost, as a result of the crisp quantization. Dubois and Prade (1990, 1992) investigated the problem of fuzzification of rough sets, and proposed rough-fuzzy and fuzzy-rough sets. Banerjee and Pal (1996) proposed a measure of roughness of a fuzzy set defined in the partitioned domain by making use of the concept of roughfuzzy sets. Banerjee and PalÕs roughness measure
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.006
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
depends on parameters that are designed as thresholds of definiteness and possibility in membership of the objects to a fuzzy set. Huynh and Nakamori (2004) introduced a parameter-free version of roughness measure for fuzzy sets based on the notion of the mass assignment of a fuzzy set. A measure of fuzziness in rough sets and some characterizations of this measure have been introduced by Chakrabarty et al. (2000). Radzikowska and Kerre (2002, 2004) defined fuzzy-rough sets dependently on fuzzy conjunction and fuzzy implication operators by applying the extension principle. Sarkar and Yagnanarayana (1998a,b) have generalized rough membership functions by defining roughfuzzy and fuzzy-rough membership functions. In this paper, we have proposed modified definition of fuzzy-rough sets by utilizing properties of fuzzy t-norm and t-conorm operators. We have proved that the modified definition is consistent with the definition proposed by Dubois and Prade (1992). However, there are a few advantages of the proposed approach. The proposed definition requires very little computational effort to perform set approximation. Further, the proposed compact computational domain identifies various patterns in the sense of fuzzy-roughness associated within given fuzzy partitioning of data and provides deeper insight to fuzzy-rough sets. The proposed definition can also be utilized to make pattern recognition algorithms faster. Various mathematical properties of this new definition have been discussed and analyzed in the sense of fuzzyroughness. Fuzzy-rough sets approach to feature selection (Bhatt and Gopal, 2004b) and fuzzy decision tree induction algorithms (Bhatt and Gopal, 2004a) are some of the significant applications of proposed fuzzy-rough sets on compact computational domain. The rest of the paper is organized as follows. Section 2 provides required theoretical background of fuzzy-rough sets, and two limiting versions of it. In Section 3 we discuss the concept of fuzzy-rough sets on compact computational domain. Various properties of this modified definition have been derived in Section 4. A brief note on the application of the proposed concepts and related references is given in Section 5. Section 6 concludes the paper.
1633
2. Theoretical foundations I = (U, P [ Q, VP, VQ, f) is a fuzzy information system, where U is non-empty finite set of objects called the universe, P & Q are non-empty finite sets of variables, P = {P1, . . . , Pj, . . . , Pp} corresponds to p inputs and Q is the output. f : U · P [ Q ! VP [ VQ is an information function such that f(U, Pj) 2 VPj; "j and f(V, Q) 2 VQ. Training patterns represented by variables P and Q are clustered into finite number of fuzzy partitions by using any of the standard fuzzy clustering algorithms (Gath and Geva, 1989; Krishnapuram and Keller, 1993; Barni et al., 1996; and Pal and Bezdek, 1995). Let U/P = {Fi}; "i, be fuzzy partitions generated by P on U using fuzzy similarity relation (Radzikowska and Kerre, 2002). Here lFi(x) = U ! [0, 1]; "x, "i, is a normal fuzzy set, i.e., maxxlFi(x) = 1 and infxmaxilFi(x) > 0. In addition to that, supxmin{lFi(x), lFj(x)} < 1; "i, j. lFi(x) is the fuzzy membership of the pattern x in the fuzzy set Fi. The set U/P builds basic partitions of knowledge about the domain of interest. Given any arbitrary fuzzy set A on the universe U, we can approximate it using basic fuzzy partitions. However, due to limited discernibility of objects, fuzzy set approximation may not be perfect. Fuzzy-rough sets are a pair of approximation i, with which we can ÔcertainlyÕ and degrees, hl; l ÔpossiblyÕ approximate arbitrary fuzzy set A; certainty approximated by l and possibility approxi. mated by l The formal definition of fuzzy-rough sets is given below. Definition 1. Given arbitrary fuzzy set lA(x) : U ! [0,1]; "x 2 U and Fi 2 U/P. According to Dubois and Prade (1990, 1992), the fuzzy- rough set is a tuple hlA ; lA i, where lower and upper approximation membership functions are defined by lA ðF i Þ ¼ inf maxf1 lF i ðxÞ; lA ðxÞg x2U
lA ðF i Þ ¼ sup minflF i ðxÞ; lA ðxÞg
ð1Þ
x2U
Two limiting versions of fuzzy-rough sets are rough-fuzzy sets and rough sets.
1634
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
1. Rough-fuzzy sets (Dubois and Prade, 1990, 1992): In this case, partitions U/P are crisp rather than fuzzy. Let, lA(x) : U ! [0,1]; "x 2 U and R is an equivalence relation (Pawlak, 1991) on U. [x]R = {y 2 UjxRy}; "x 2 U. Rough-fuzzy set is an approximation of fuzzy set A on crisp approximation space U/P. lA ð½xR Þ ¼ infflA ðxÞjx 2 ½xR g lA ð½xR Þ ¼ supflA ðxÞjx 2 ½xR g
ð2Þ
2. Rough sets (Pawlak, 1991): Let, lA(x) : U ! {0,1}; "x 2 U and R is an equivalence relation on U. Rough set is an approximation of crisp set A on crisp approximation space U/P. A ¼ fx 2 U j ½xR Ag ¼ fx 2 U j ½x \ A 6¼ ;g A R
ð3Þ
where A is called the lower approximation and A is called the upper approximation of A. 3. Proposed fuzzy-rough sets on compact computational domain In this section, we propose the concept of fuzzyrough sets on compact computational domain by utilizing properties of fuzzy t-norm and t-conorm operators. 3.1. Fuzzy logical operators Definition 2 (Radzikowska and Kerre, 2002). Fuzzy t-norm (T) and t-conorm (S) are increasing, associative and commutative mapping 2
2
T : ½0; 1 ! ½0; 1 and S : ½0; 1 ! ½0; 1 that satisfies the boundary conditions: T ðx; 1Þ ¼ x; T ðx; 0Þ ¼ 0; 8x 2 U Sðx; 0Þ ¼ x; Sðx; 1Þ ¼ 1; 8x 2 U
S n ða1 ; . . . ; an1 ; an Þ ¼ SðS n1 ða1 ; . . . ; an1 Þ; an Þ
3.2. Modified fuzzy-rough sets Using neutral properties of fuzzy logic operators, we suggest new definitions of lower and upper approximation membership functions. Definition 3. Given arbitrary fuzzy set lA(x) : U ! [0,1]; "x 2 U and Fi 2 U/P. The fuzzy-rough set on compact computational domain is a tuple hlA ; lA i for x 2 DA(Fi) and x 2 DA ðF i Þ respectively. Lower and upper approximation membership functions are defined by 8 inf maxf1 lF i ðxÞ; lA ðxÞg; > > > < x2DA ðF i Þ lA ðF i Þ ¼ DA ðF i Þ 6¼ ; > > > : 1; DA ðF i Þ ¼ ; 8 < sup minflF i ðxÞ; lA ðxÞg; DA ðF i Þ 6¼ ; lA ðF i Þ ¼ x2DA ðF i Þ : 0; DA ðF i Þ ¼ ; ð5Þ where DA(Fi) U and DA ðF i Þ U are compact computational domains for lower and upper approximation membership functions: DA ðF i Þ ¼ fx 2 U jlF i ðxÞ 6¼ 0 ^ lA ðxÞ 6¼ 1g DA ðF i Þ ¼ fx 2 U jlF i ðxÞ 6¼ 0 ^ lA ðxÞ 6¼ 0g
ð6Þ
Here ‘‘^’’ is logical AND connective. ð4Þ
Due to their associative properties we can extend them to nary operation as, T n ða1 ; . . . ; an1 ; an Þ ¼ T ðT n1 ða1 ; . . . ; an1 Þ; an Þ
nary operators Tn and Sn, which are the extension of triangular norm and conorm to n arguments, satisfy similar properties as the original binary T and S. The most popular t-norm and t-conorm used in fuzzy reasoning approaches are standard ‘‘min’’ and ‘‘max’’ operators, respectively. For this family of fuzzy t-norm operators, ‘‘one’’ is the neutral element and for fuzzy t-conorm operators, ‘‘zero’’ is the neutral element.
By Eq. (6) DA ðF i Þ ¼ fi \ AC DA ðF i Þ ¼ fi \ AS
ð7Þ
where AC = {x 2 UjlA(x) = 1}, AS = {x 2 UjlA(x) > 0}, and fi = {x 2 UjlFi(x) > 0}. It is defined as fuzzy-rough set on compact computational domain because it suffices to
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
calculate lower and upper approximation memberships only on certain compact domain, defined in Eq. (7), instead of all x 2 U. The definition given above is consistent with that given in Eq. (1) but requires less computational effort for calculating lower and upper approximations. For fuzzy t-norm operators, ‘‘one’’ is the neutral element. The region where max{1lFi(x),lA(x)} = 1 does not have any impact on the formation of lower approximation membership due to ÔinfÕ operator on it. We will find a region where max{1lFi(x), lA(x)} 5 1, and compute lower and upper approximation membership degrees only on this region. If this region is empty we will show that lA(Fi) = 1. Let maxf1 lF i ðxÞ; lA ðxÞg 6¼ 1 ) lF i ðxÞ 6¼ 0 ^ lA ðxÞ 6¼ 1;
x2U
) x 2 f i \ AC It is obvious that iff DA(Fi) = ;, then 9 = x 2 U, 3max{1 lFi(x), lA(x)} 5 1, i.e., all the patterns x 2 Fi, belong to the core of fuzzy set A. In this case, fi AC and lA(Fi) = 1. The consistency and compactness of upper approximation definition can be proved in a similar way. If DA ðF i Þ ¼ ;, then Fik and A do not have any pattern in common (i.e., fi \ AS = ;). Further, if DA(Fi) = U, then lA(Fi) never becomes 1. Let DA ðF i Þ ¼ U ) fi \ AC ¼ U ) fi ¼ U ^ AC ¼ U ) fi ¼ U ^ AC ¼ ; For lA(Fi) to be 1, max{1 lFi(x), lA(x)} = 1; "x 2 U. Now, lA(x) 5 1 ({AC = ;) and lFi(x) 5 0 ({fi = U). Thus lA(Fi) never becomes 1 if DA(Fi) = U. Qualitatively, fi = U ^ AC = ; indicates a situation where no elements belong to the core of fuzzy set A, and support(Fi) = U. fi = U is possible, though rarely, but AC = ; never happens in practice. When the classification is crisp (i.e., A is a crisp set), where either an element belongs to the set or not (i.e., crisp partitioning of decision region); saying that the core of a certain classification is empty is equivalent to saying that the clas-
1635
sification does not exist at all. When A is fuzzy, there will be some training patterns which belong to the core of A. So, for real world applications, compact computational domain is strictly a subset of U. DA ðF i Þ U and DA ðF i Þ < n This shows that the number of elements on which calculations are required by the proposed definition will be always less than the total number of training patterns. To explain the computations of the proposed domains, a small example dataset (Table 1) has been taken from Jensen and Shen (2002). Q classifies each of the objects to either of the classes: ÔNoÕ or ÔYesÕ. Two equivalence classes formed by Q are U/Q = {No, Yes} = {[1, 3, 6], [2, 4, 5]}. These two classes can be thought of as fuzzy sets, with the patterns belonging to the class possessing a membership of one, zero otherwise. The variable P1 has been partitioned into two fuzzy sets shown in Fig. 1. Let us consider the approximation of equivalence classes formed by ÔNoÕ and ÔYesÕ with fuzzy set F1. f1 ¼ fx 2 U jlF 1 ðxÞ > 0g ¼ f1; 2; 3g
Table 1 Example dataset Pattern
P1
P2
P3
Q
1 2 3 4 5 6
0.4 0.4 0.3 0.3 0.2 0.2
0.3 0.2 0.4 0.3 0.3 0
0.5 0.1 0.3 0 0 0
No Yes No Yes Yes No
Fig. 1. Fuzzy sets for P1.
1636
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
By the proposed definition, lower and upper approximation compact computational domains for the equivalence class formed by ÔNoÕ are defined by DNo ðF 1 Þ ¼ f1 \ f1; 3; 6g ¼ f1; 2; 3g \ f2; 4; 5g ¼ f2g DNo ðF 1 Þ ¼ f1 \ f1; 3; 6g ¼ f1; 2; 3g \ f1; 3; 6g ¼ f1; 3g This shows that only pattern {2} belongs to the lower approximation and patterns {1, 3} belong to the upper approximation effective regions. Calculating lower and upper approximation membership degrees only on this compact region, we get the following: lNo ðF 1 Þ ¼ inf maxf1 0:8; 0g ¼ inff0:2g ¼ 0:2 lNo ðF 1 Þ ¼ supfminf0:8; 1g; minf0:6; 1gg ¼ supf0:8; 0:6g ¼ 0:8 This example shows that the proposed definition identifies the effective region for the formulation of lower and upper approximation and then calculates membership degrees only on this compact domain rather than calculating it for all the patterns. For larger databases, it will result in high computational gain. 4. Properties of proposed version of fuzzy-rough sets As an outgrowth of presented definition of fuzzy-rough sets, here are some of the qualitative and quantitative observations which can further explore the significance of the proposed approach from pattern classification viewpoint. Observation 1. DA ðF i Þ ¼ ; ¼ DA ðF i Þ if and only if no fuzzy-roughness is associated with all the patterns x, "x 2 Fi. Proof. No fuzzy-roughness is associated with pattern x if fuzzy set Fi in which x has non-zero membership value belongs only to one class A, or does not share any pattern with the class A. Let DA ðF i Þ ¼ ; () fi \ AC ¼ ; () fi AC () lA ðF i Þ ¼ 1
This way, all the patterns from fuzzy set Fi completely belong to the core of A; fuzzy set Fi perfectly approximates arbitrary fuzzy set A. By the same logic one can show that with DA ðF i Þ ¼ ;, 9 = x, 3(x 2 fi ^ x 2 AS), i.e., fi and AS do not have any pattern in common and thus lA ðF i Þ ¼ 0; fuzzy set Fi completely does not approximate arbitrary fuzzy set A. h Observation 2. If no fuzzy linguistic uncertainty is associated with the pattern x, "x 2 U, then the proposed definition will be converted to the definition of rough-fuzzy lower and upper approximation memberships on respective compact computational domains. Proof. If no fuzzy linguistic uncertainty is associated with pattern x, "x 2 U, then each partition is crisp with lFi(x) : U ! {0,1} and let, lA(x) : U ! [0,1]; "x 2 U. Fi can be regarded as an equivalence class [x]R, where [x]R = {y 2 UjxRy}; "x 2 U 8 < inf flA ðxÞg; ½xR \ AC 6¼ ; lA ð½xR Þ ¼ x2½xR \AC : 1; ½xR \ AC ¼ ; 8 < sup flA ðxÞg; ½xR \ AS 6¼ ; lA ð½xR Þ ¼ x2½xR \AS : 0; ½xR \ AS ¼ ; It is clear from above definition that lA([x]R) = 0 if and only if [x]R X AS, i.e., [x]R and A do not have any pattern in common. lA ð½xR Þ ¼ 1 if and only if [x]R \ AC 5 ;, i.e., at least one element from equivalence class [x]R belongs to the core of fuzzy set A. h Observation 3. If no fuzzy linguistic and fuzzy classification uncertainties are associated with all the patterns x 2 U, then, lFi(x) : U ! {0,1} and lA(x) : U ! {0,1}; "x 2 U. For this case the proposed definition reduces to classical rough set presentation with corresponding compact domain. ( 1; ½xR A lA ð½xR Þ ¼ 0; ½xR 6 A ( 1; ½xR \ A 6¼ ; lA ð½xR Þ ¼ 0; ½xR \ A ¼ ;
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
1637
The definitions of lower and upper compact computational domains in this case becomes equal, as the set which is to be approximated is crisp, i.e., AC = AS = A. Thus, DA ð½xR Þ ¼½xR \ A ¼ DA ð½xR Þ.
DA ðF 1 ; . . . ; F j Þ ¼ f1 \ \ fj \ AC
Observation 4. If the partitioning is fine, then the proposed computational domain identifies the region where degree of belongingness of the pattern x to fuzzy set A is equal to one, greater than zero but not equal to one, and equal to zero.
Proof. To prove this we will find a region where max{1 min(lF1(x), . . . , lFj(x)), lA(x)} 5 1; and define compact computational domain by set theoretic difference of U with identified region.
Proof. When the partitioning is fine, each pattern x is being treated individually rather then belonging to a fuzzy or crisp partitions. The membership degrees of such fine partitions are thus represented here by l(x) only and are equal to 1. Lower and upper approximation membership degrees of fuzzy set A by such partitions are denoted by lA(x) and lA ðxÞ for all x 2 U. For this formulation, it can be easily proved that lA ðxÞ ¼ lA ðxÞ ¼ lA ðxÞ. From the proposed definition DA ðxÞ ¼ ; () x \ AC ¼ ;; i:e:; x 2 AC () lA ðxÞ ¼ 1: DA ðxÞ ¼ ; () x \ AS ¼ ;; i:e:;
x 62 AS () lA ðxÞ ¼ 0:
If DA 5 ; and DA 6¼ ;, x 62 AC and x 2 AS. This implies that, 0 < lA(x) < 1. h
¼ DA ðF 1 Þ \ \ DA ðF j Þ DA ðF 1 ; . . . ; F j Þ ¼ f1 \ \ fj \ AS ¼ DA ðF 1 Þ \ \ DA ðF j Þ
maxf1 minðlF 1 ðxÞ; . . . ; lF j ðxÞÞ; lA ðxÞg 6¼ 1; ) minðlF 1 ðxÞ; . . . ; lF j ðxÞÞ 6¼ 0 ^ lA ðxÞ 6¼ 1; 8x 2 U ) ðlF 1 ðxÞ ^ ^ lF j ðxÞ 6¼ 0Þ ^ ðlA ðxÞ 6¼ 1Þ; 8x 2 U ) x 2 f1 \ \ fj \ AC Thus DA(F1, . . . , Fj) = DA(F1) \ \ DA(Fj). Proof for DA ðF 1 ; . . . ; F j Þ can be developed on similar lines. h Note that DA ðF 1 ; F 2 ; . . . ; F j Þ DA ðF i Þ; i ¼ 1; . . . ; j jDA ðF 1 ; F 2 ; . . . ; F j Þj 6 DA ðF i Þ; i ¼ 1; . . . ; j From this observation, the following two results can be developed trivially. 1. DA(F1 [ F2) = DA(F1) [ DA(F2) 2. DA ðF 1 [ F 2 Þ ¼ DA ðF 1 Þ [ DA ðF 2 Þ
To compute lower approximation using more than one variable, Cartesian product of fuzzy partitions generated by given set of variables should be considered. In this case
We will prove the first; the second can be derived on similar lines. 1. lA ðF 1 [ F 2 Þ ¼ inf x2U maxf1 maxðlF 1 ðxÞ; lF 2 ðxÞÞ; lA ðxÞg. Proceeding as in Observation 5.
lfF 1 ;...;F j g ðxÞ ¼ minflF 1 ðxÞ; . . . ; lF j ðxÞg
maxf1 maxðlF 1 ðxÞ; lF 2 ðxÞÞ; lA ðxÞg 6¼ 1
and
() ðlF 1 ðxÞ 6¼ 0 _ lF 2 ðxÞ 6¼ 0Þ ^ lA ðxÞ 6¼ 1; x 2 U
lA ðF 1 ; . . . F j Þ 8 inf maxf1 minðlF 1 ðxÞ; . . . ; lF j ðxÞÞ; > > < x2DA ðF 1 ;...;F j Þ ¼ lA ðxÞg; DA ðF 1 ; . . . ; F j Þ 6¼ ; > > : 1; DA ðF 1 ; . . . ; F j Þ ¼ ;
() x 2 ðf1 [ f2 Þ \ AC
Observation 5. Compact computational domain for set approximation with Cartesian product of fuzzy partitions can be defined as,
() x 2 DA ðF 1 Þ [ DA ðF 2 Þ Observation 6 1. 2. 3. 4.
DA[B(Fi) = DA(Fi) \ DB(Fi). DA\B(Fi) = DA(Fi) [ DB(Fi). DA[B ðF i Þ ¼ DA ðF i Þ [ DB ðF i Þ. DA\B ðF i Þ ¼ DA ðF i Þ \ DB ðF i Þ.
1638
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
Proof. We will prove the first and the fourth; the other two can be proved on similar lines of argument. 1. lA[B ðF i Þ ¼ inf maxf1 lF i ðxÞ; maxðlA ðxÞ; lB ðxÞÞg x2U
We will find a region where max{1 lFi(x), max(lA(x), lB(x))} 5 1. maxf1 lF i ðxÞ; maxðlA ðxÞ; lB ðxÞÞg 6¼ 1 () lF i ðxÞ 6¼ 0 ^ ðlA ðxÞ 6¼ 1 ^ lB ðxÞ 6¼ 1Þ; 8x 2 U () x 2 fi \ AC \ BC () x 2 DA ðF i Þ \ DB ðF i Þ 4. lA\B ðF i Þ ¼ sup minflF i ðxÞ; minðlA ðxÞ; lB ðxÞÞg x2U
We will find a region where min{lFi(x), min(lA(x), lB(x))} 5 0. minflF i ðxÞ; minðlA ðxÞ; lB ðxÞÞg 6¼ 0 () lF i ðxÞ 6¼ 0 ^ ðlA ðxÞ 6¼ 0 ^ lB ðxÞ 6¼ 0Þ; 8x 2 U () x 2 fi \ ðAS \ BS Þ () x 2 DA ðF i Þ \ DB ðF i Þ
Following is an interesting observation on the basis of the above two propositions. Observation 7. Observations 5 and 6 can be combined together to define fuzzy rough sets on compact computational domain for approximating multiple fuzzy sets involving intersection and union operations. For example, two fuzzy sets A and B, when approximated through two fuzzy sets F1 and F2, the compact computational domain can be obtained as, DA\B ðF 1 ; F 2 Þ ¼ DA ðF 1 ; F 2 Þ [ DB ðF 1 ; F 2 Þ ¼ fDA ðF 1 Þ \ DA ðF 1 Þg
H X 1 lF j ðxÞ ljA j¼1 lF j ðxÞ j¼1
T fA ðxÞ ¼ PH where ljA
kA \ F j k ¼ ¼ kF j k
P
x2U
minðlF j ðxÞ; lA ðxÞÞ P x2U lF j ðxÞ
T fA ðxÞ is constrained fuzzy-rough membership function defined by Sarkar and Yagnanarayana (1998a) and H is the number of fuzzy sets in which x has non-zero membership value. Proof. T fA ðxÞ ¼ 1 if and only if ljA ¼ 1; "j = 1, . . . , H. This condition implies that Fj A; "j = 1, . . . , H. In Observation 1 we have proved that Fj AC ) Fj AS iff DA(Fj) = ;; "j. This in turn implies that, ljA ¼ 1; 8j ¼ 1; . . . ; H . This completes the proof. This proposition shows that if DA(Fj) = ;, then fuzzy-roughness is absent for all the patterns x 2 Fj. In the same way, one can show that T fA ðxÞ ¼ 0 ) ljA ¼ 0; "j = 1, . . . , H. In Observation 1 we have also shown that, if DA ðF j Þ ¼ ;; "j = 1, . . . , H, then Fj; "j = 1, . . . , H and A do not have any pattern in common. This indicates that, ljA ¼ 0; "j = 1, . . . , H. h Observation 9. From the definition of compact computational domain of lower and upper approximation memberships, the classification properties of fuzzy set Fi, in classifying an arbitrary fuzzy set A can be grouped into three basic categories: 1. A is completely Fi-observable in a fuzzy-rough manner iff DA(Fi) = ;. 2. A is partially Fi-observable in a fuzzy-rough manner iff DA(Fi) 5 ; and DA ðF i Þ 6¼ ;. 3. A is completely Fi-unobservable in a fuzzyrough manner if DA ðF i Þ ¼ ;.
[ fDB ðF 1 Þ \ DB ðF 2 Þg Observation 8 1. DA ðF i Þ ¼ ; ) T fA ðxÞ ¼ 1; "x 2 Fi 2. DA ðF i Þ ¼ ; ) T fA ðxÞ ¼ 0; "x 2 Fi
Defined compact computational domain for fuzzy-rough sets has close link with minima of classification entropy (Umano et al., 1994) and classification ambiguity (Yuan and Shaw, 1995) functions.
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
Let, with the given fuzzy set Fi, q discrete classes of concern, i.e., {1, 2, . . . , q} are required to classify. Consider two functions; q X pil log2 pil and Entri ¼
gðpi1 ; pi2 ; . . . ; piq Þ ¼
q X l¼2
1639
pil ln
l l1
It is easy to verify that g gets its minimum only at (1, 0, . . . , 0). h
l¼1 q X pil pi;lþ1 ln l; Ambgi ¼
5. Applications
l¼1
where Entri and Ambgi are classification entropy and classification ambiguity of Fi, respectively. pil is the certainty of Fi concerning lth class defined as, P minðlF i ðxÞ; ll ðxÞÞ P pil ¼ x2U x2U minðlF i ðxÞÞ pil ; "l = 1, . . . , q, and p ; . . . ; p pil ¼ max16l6q are i1 iq pil descending order arrangements of (pi1, . . . , piq). Observation 10. If $l 2 Q, 3Dl(Fi) = ;, function Entri and Ambgi attain their minimum simultaneously. Proof. From Observation 1, if $l 2 Q, 3Dl(Fi) = ;, then Fi l and Fi \ j = ;; "j 2 Q, j 5 l. It indicates the situation where Fi approximates one class perfectly, and does not share any pattern with other classes. From the definition of pil, it is self-evident that {pi1, . . . , pil, . . . , pij, . . . , piq} = {0, . . . , 1, . . . , 0, . . . , 0}. Now, with an appointment 0 · log20 = limx!0(x · log2x) = 0, function Entri attains its minimum at a vector of certainty factors where each component is either 0 or 1, and Ambgi attains its minimum at a vector in which only one component is 1 and all other components are 0. To prove this, consider the second derivative of function Entri. ! q X o2 1 <0 pil log2 pil ¼ p op2il ln 2 il l¼1 This implies that Entri is convex function concerning pil 2 (0,1]. A convex function attains its minimum at the extremes. To prove for the function Ambg Pq i, consider the function gðpi1 ; pi2 ; . . . ; piq Þ ¼ with an appointment l¼1 ðp il p i;lþ1 Þ ln l {(pi1, . . . , piq)j1 = pi1 P P piq = 0}, without loss of generality. Function g can be written as:
5.1. Fast fuzzy-rough feature selection (Bhatt and Gopal, 2004b) The proposed concept of fuzzy-rough sets on compact computational domain has been applied to speed-up the tree structured feature selection algorithm FRSAR (fuzzy-rough set attribute reduction) (Jensen and Shen, 2002). Applying Observation 5 on each level of the search tree, we have achieved computational gain as high as 622 compared to FRSAR. 5.2. Weighted fuzzy classification rules induction (Bhatt and Gopal, 2004a) Fuzzy decision tree induction algorithm for the extraction of weighted fuzzy classification rules for medical informatics problems has been developed. The concept of compact computational domain has been utilized for pattern pruning at each subsequent level of the decision tree to accelerate the computation. 6. Concluding remarks In this paper, we have proposed a modified definition of fuzzy-rough sets by identifying computational domains based on natural properties of fuzzy t-norms and t-conorms. We have discussed some mathematical properties of fuzzy-rough sets on compact computational domain from pattern classification viewpoint. It is established that in addition to providing deeper insight to fuzzyrough sets, the new definition identifies various patterns in the sense of fuzzy-roughness and can also be utilized to make pattern recognition algorithms faster. Though the concept of shadowed sets proposed by Pedrycz (1998) looks similar to one proposed by us, however, there is a
1640
R.B. Bhatt, M. Gopal / Pattern Recognition Letters 26 (2005) 1632–1640
fundamental difference. In fuzzy-rough sets, the approximation space is defined in advance and the equivalence classes are kept fixed. The proposed compact computational domains have been formulated with respect to these equivalence classes. In the concept of shadowed sets, these classes are searched dynamically. As a further research, one can think of a certain algorithmic generalization by borrowing concepts from both to develop a model of uncertainty. Acknowledgement The authors are thankful to the anonymous referees for their helpful and precious comments. References Banerjee, M., Pal, S.K., 1996. Roughness of a fuzzy set. Informat. Sci. 93, 235–246. Barni, M., Cappellini, V., Mecocci, A., 1996. Comments on a possibilistic approach to clustering. IEEE Trans. on Fuzzy Systems 4 (3), 393–396. Bhatt, R.B., Gopal, M., 2004a. Induction of weighted and interpretable fuzzy classification rules for medical informatics. In: Proc. Internat. Conf. on System., Cybernet. Informat. (ICSCI 2004), February 12–15, Hyderabad, India, pp. 371–376. Bhatt, R.B., Gopal, M., 2004b. On fuzzy-rough sets approach to feature selection. Pattern Recognition Lett. (in press), doi:10.1016/j.patrec.2004.09.044. Chakrabarty, K., Biswas, R., Nanda, S., 2000. Fuzziness in rough sets. Fuzzy Sets Syst. 110, 247–252. Dubois, D., Prade, H., 1990. Rough fuzzy sets and fuzzy rough sets. Internat. J. Gener. Syst. 17 (2–3), 191–209. Dubois, D., Prade, H., 1992. Putting fuzzy sets and rough sets together. In: Slowinski, R. (Ed.), Intelligent Decision Support. Kluwer Academic, Dordrecht.
Gath, I., Geva, A.B., 1989. Unsupervised optimal fuzzy clustering. IEEE Trans. Pattern Anal. Machine Intell. 11 (7), 773–781. Huynh, V.N., Nakamori, Y., 2004. An approach to roughness of fuzzy sets. In: Proc. IEEE Internat. Conf. on Fuzzy Syst., July 25–29. Krishnapuram, R., Keller, J.M., 1993. A possibilistic approach to clustering. IEEE Trans. Fuzzy Syst. 1 (2), 98–110. Pal, N.R., Bezdek, J.C., 1995. On cluster validity for the fuzzy c-means model. IEEE Trans. Fuzzy Syst. 3 (3), 370–379. Pal, S.K., Skowron, A. (Eds.), 1999. Rough Fuzzy Hybridization: A New Trend in Decision Making. Springer-Verlag, Berlin. Pawlak, Z., 1991. Rough Sets Theoretical Aspects of Reasoning about Data. Kluwer, Norwell, MA. Pawlak, Z., 1999. Rough classification. Internat. J. Human Comput. Stud. 51 (2), 369–383. Pawlak, Z., 2002a. Rough sets and intelligent data analysis. Inform. Sci. 147, 1–12. Pawlak, Z., 2002b. Rough sets, decision algorithms, and BayeÕs theorem. Europ. J. Operation. Res. 136 (1), 181–189. Pedrycz, W., 1998. Shadowed sets: Representing and processing fuzzy sets. IEEE Trans. SMC-B 28 (1), 102–109. Radzikowska, A.M., Kerre, E.E., 2002. A comparative study on fuzzy-rough sets. Fuzzy Sets Syst. 126, 137–155. Radzikowska, A.M., Kerre, E.E., 2004. An algebraic characterization of fuzzy rough sets. In: Proc. IEEE Internat. Conf. on Fuzzy Syst., July 25–29. Sarkar, M., Yagnanarayana, B., 1998a. Fuzzy-rough membership functions. In: Proc. IEEE Internat. Conf. on SMC, October 11–14, pp. 2028–2033. Sarkar, M., Yagnanarayana, B., 1998b. Rough-fuzzy membership functions, In: Proc. of IEEE Internat. Conf. on Fuzzy Syst., May 4–9, pp. 796–801. Slowinski, R. (Ed.), 1992. Intelligent Decision Support. Handbook of Applications and Advances of the Rough Sets Theory. Kluwer Academic Publishers, Dordrecht. Umano, M., et al., 1994. Fuzzy decision tree by fuzzy ID3 algorithm and its application to diagnosis systems. In: Proc. of IEEE Internat. Conf. on Fuzzy Syst., June 26–29, pp. 2113–2118. Yuan, Y., Shaw, M.J., 1995. Induction of fuzzy decision trees. Fuzzy Sets Syst. 69, 125–139.
Pattern Recognition Letters 26 (2005) 1710–1719 www.elsevier.com/locate/patrec
Color texture analysis using the wavelet-based hidden Markov model Qing Xu *, Jie Yang, Siyi Ding Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Box 251, 1954 Huashan Road, Shanghai 200030, China Received 24 September 2004; received in revised form 24 September 2004 Available online 18 April 2005 Communicated by E. Backer
Abstract The wavelet-domain hidden Markov model (WD HMM), in particular the hidden Markov tree (HMT), has recently been proposed and applied to gray texture analysis with encouraging results. For color texture analysis, the previous WD HMM can only be used to model the different color planes individually, assuming they are independent of each other. However, this assumption in general is unrealistic. We show in this paper that the wavelet coefficients have certain inter-dependences between color planes. This paper presents a novel approach to modeling the dependences between color planes as well as the interactions across scales. In our approach, the wavelet coefficients at the same location, scale and sub-band, but different color planes are grouped into one vector. We then propose a multivariate Gaussian mixture model (MGMM) for approximating the marginal distribution of the wavelet coefficient vectors in one scale and capturing the interactions of different color planes. In addition, the statistical dependence between different scales is captured by the transition matrix of the hidden Markov tree. Using this approach, we can improve the performance of the WD HMM on the color texture classification. The experiments show that our WD HMM approach provides 85% of correct classifications (PCC) on 68 color textures from Oulu texture database and outperforms the other wavelet-based methods we study. In this paper, we also investigated the classification performance of the WD HMM methods on different color spaces. 2005 Elsevier B.V. All rights reserved. Keywords: Wavelet-domain hidden Markov model; Color texture analysis; Multivariate Gaussian mixture model
1. Introduction *
Corresponding author. Tel./fax: +86 2162933552. E-mail address:
[email protected] (Q. Xu).
Texture and color are of considerable importance for image analysis, especially when we study
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.013
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
and process real world images. In fact, varieties of texture models have been discussed in the literature intensively (Krishnamachari and Chellappa, 1997; Tardif and Zaccarin, 1997; Fan and Xia, 2000a,b; Laine and Fan, 1993). But most of them are based on gray level images, ignoring color information. On the other hand, the description of color mainly involves color histograms and does not consider spatial texture information (Tan and Kittler, 1993; Wan and Kuo, 1996). In this paper, we present an improved wavelet-domain hidden Markov model for jointly modeling texture and color of images. The wavelet-domain hidden Markov tree (WD HMT) model has recently been proposed by Crouse et al. (1998). This method has been successfully applied to image texture analysis, because it can effectively characterize the joint statistics of the wavelet transforms by capturing the inter-scale dependences of the wavelet coefficients. But all the applications of the WD HMT so far are limited to gray textures. For color texture analysis, the previous WD HMT can only be used to model color planes separately. If this method works, it must assume that different color planes are independent of each other. In fact, however, we have observed that the regular and homogeneous structures of different color planes often manifest certain dependencies between different color components and thus we must model this kind of dependencies as well as inter-scale interactions. To solve this problem, we have developed an improved version of the WD HMT. The wavelet coefficients at the same location, scale and subband, but different color planes are grouped into one vector. We then propose a multivariate Gaussian mixture model (MGMM) for approximating the marginal distribution of wavelet coefficient vectors in one scale, and the MGMM can also describe the interactions of different color planes. Just like the previous WD HMT (Crouse et al., 1998), the hidden state variables for each wavelet vector are linked across scales to build a tree structure. Thus, the statistical dependence between different scales is captured by the transition matrix of the hidden Markov tree. Obviously our method captures not only the inter-scale dependences but also the interactions between different color com-
1711
ponents. In this paper, we also study the choice of color representation space, which has a great effect on the performance of a color analysis method. We show how our method works in both the RGB and K–L (Karhunen Loeve transform) or I1I2I3 (Ohta, 1985) color space to evaluate the choice of the color space. In the experiments, we also compare our work with three other wavelet-based analysis methods, focusing on performance evaluation. The first (Van de Wouwer et al., 1998) uses the wavelet energy correlation signature. This method models the statistics of color textures less completely than our method. The second applies the WD HMT based on gray images to each color plane individually. This would lead to the loss of dependence information between different color planes, while our method can capture this dependence. The third adopts the WD HMT based on gray images to characterize the gray level images of color textures. This approach fails to incorporate the color information into the model. The first method adopts a k-nearest neighbor (KNN) classifier, while the other two are both based on the maximum likelihood (ML) classifier. We demonstrate that our method provides the best percentage of correct classification (PCC) upon 68 color textures from Oulu texture database. In particular, we evaluate the performance of our method in both the RGB and the K–L (Karhunen Loeve transform) or I1I2I3 color space. This paper is organized as follows. In Section 2, we review the previous work related to the WD HMT methods. In Section 3, we develop a novel WD HMT approach to characterize color textures accurately by involving a more sophisticated graph structure. In Section 4, we show how this model can be applied to color texture classification. We compare our model with other wavelet-based methods and show the high-performance of our model. In Section 5, we present and discuss the simulation results. Finally, Section 6 summarizes the paper.
2. Related works Crouse et al. (1998) has recently proposed a framework for statistical signal modeling based
1712
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
on the wavelet-domain hidden Markov tree (WD HMT). It provides an effective approach to modeling both the non-Gaussianity1 and the persistence2 of wavelet coefficients that are often found in real world images. In this section, we briefly review the basic concept of the 2D WD HMT. Here we denote wBj;i;k to be the wavelet coefficient of the 2D wavelet transform of a gray image. In this representation, j indexes scale (1 < j < J)— smaller j corresponds to higher resolution analysis. (i, k) indexes the spatial location of this wavelet coefficient in the j scale. B represents sub-band detail image (B 2 {HH, HL, LH}). The WD HMT employs a Gaussian mixture model (GMM) to model the non-Gaussian wavelet coefficients marginal statistics. Each wavelet coefficient wBj;i;k is associated with a discrete hidden state random variable S Bj;i;k , whose probability mass function can be expressed as P ðS Bj;i;k ¼ mÞ ¼ ps ðmÞ (m = 0 M 1). Conditioning on its state S Bj;i;k ¼ m, wBj;i;k follows a Gaussian distribution with mean lm and variance rm. This Gaussian mixture closely matches the wavelet coefficient marginal distribution observed in natural images. Although we can approximate the real distributions arbitrarily close (Crouse et al., 1998) by increasing M and allowing non-zero means, we set M = 2 and lm = 0 for simplicity in this paper. Therefore, we parameterize the GMM of wBj;i;k by p ¼ fps ðmÞ; r2m jm ¼ 0; 1g and the overall probability density function of wBj;i;k is expressed as fX ðxÞ ¼
1 X
P S ðmÞfX jS ðxjs ¼ mÞ;
ð1Þ
Second, a tree model is constructed in each subband separately to capture the persistence of the wavelet coefficients. By connecting the hidden state (parent) at a coarse scale to the four hidden states (children) at the next intermediate fine scale, we obtain a tree-structured graph for each sub-band (Fig. 1). Thus, a state transition matrix for each parent ! child link statistically quantifies the persistence of wavelet coefficients. Furthermore, in order to obtain a robust estimation, the wavelet coefficients in the same scale and sub-band are assumed to share the same statistics (GMM and transition matrix), which is called tying (Crouse et al., 1998). Thus, when a J scale wavelet transform is preformed on an image, the 2-D HMT (M = 2, zero mean) consists of three independent tree models and can be parameterized by hHMT
gray
¼ fpBJ ðmÞ; eBj;j1 ðm; nÞ; r2B;j;m jB 2 fHH ; HL; LH g; j ¼ 1; . . . ; J ;m; n ¼ 0; 1g; where eBj;j1 ðm; nÞ is the state transition probability of the Markov chain from scale j to scale (j 1) in the sub-band B and P BJ is the probability mass function of the hidden state random variable of the J scale wavelet coefficients in the sub-band B. hHMT_gray can be fitted to a given wavelet transform of an image W by the tree-structured EM training algorithm, which aims at maximizing the posterior likelihood f(WjhHMT_gray). Due to its effective characterization of the statistics of images, the WD HMT is widely adopted in
m¼0
where 1 x2 fX jS ðxjs ¼ mÞ ¼ pffiffiffiffiffiffiffiffiffiffiffi exp 2 ¼ gðx; r2m Þ. 2rm 2pr2m ð2Þ Here gðx; r2m Þ denotes the zero mean Gaussian distribution with variance rm.
1
Non-Gaussainity: the wavelet coefficients have a peaky, heavy-tailed marginal distribution. 2 Persistence: the large/small values of wavelet coefficients tend to propagate through the scales of the wavelet transform.
Fig. 1. 2-D hidden Markov tree structure, where the black nodes represent the hidden state and the white nodes represent the wavelet coefficient associated with that hidden state and the links between the black nodes capture the dependences across scales.
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
the image processing area. Choi and Baraniuk (2001) has combined a WD HMT based texture classification and a multi-scale fusion to segment texture images. Fan and Xia (2000a,b) has applied the WD HMT to texture analysis. Romberg et al. (2001) has used the WD HMT as a means of image de-noising. Moreover, Fan and Xia (2000a,b) and Do and Vetterli (2002a,b) have improved the hidden Markov tree model by completely considering the cross-correlation of the different sub-bands (HH, HL, LH). However, all their works only focus on the gray level images. In this paper, we have developed an improved WD HMT for modeling color images.
3. Wavelet-domain hidden Markov model for color images The most straightforward extension of the 2D WD HMT to color images is to transform each color plane and then model each transformed color plane independently with the 2D WD HMT introduced in Section 2. Given a color image I composed of three color component images (I X 1 ; I X 2 ; I X 3 ), we can obtain their respective 2D HMT parameters X3 X2 1 (hXHMT gray ; hHMT gray ; hHMT gray ) by substituting I X 1 , I X 2 and I X 3 consecutively for the gray image in the modeling procedure in Section 2. Here Xm {m = 1, 2, 3} represent the color components based on a certain color space. The likelihoodP of I belonging to this model can be expressed as A2fX 1 ;X 2 ;X 3 g logðf ðW A jhAHMT gray ÞÞ, where WA denotes the wavelet coefficients of IA (A 2 {X1, X2, X3}). The underlying assumption for this approach is that the three color planes are independent of each other. However, some previous authors (Caeli and Reye, 1993) have pointed out that the correlations existing between different color planes determine the color texture perceived by humans. So it is possible to improve the accuracy of color texture characterization by taking account of the cross-correlations of different color planes. To incorporate the dependencies between different color planes, we group the wavelet coefficients at the same location, scale and sub-band, but different color planes into one vector and then model
1713
the vectors in the same sub-band by one quad-tree structure. More specifically, we denote the wavelet coefficient of A component (A 2 {X1, X2, X3}) at sub-band B (B 2 {HH, LH, HL}), located at (k, i) in j scale as wAB;k;i;j . As described in Section 2, wAB;k;i;j is associated with a hidden state random 1 ; variable S AB;k;i;j . Then, three coefficients wXB;k;i;j X3 X2 wB;k;i;j ; wB;k;i;j are grouped to generate one vector, T 3 1 2 ~ wB;k;i;j ¼ fwXB;k;i;j ; wXB;k;i;j ; wXB;k;i;j g .
We assume that the distribution density of the wavelet coefficient vectors is approximated by a three-dimensional multivariate Gaussian mixture model (MGMM). The justification for this is that, since any marginal density from a multivariate Gaussian density is also a Gaussian density, the marginal density from an MGMM is also a GMM. Thus, we can expect that the MGMM also captures the marginal distribution densities of the three dependent color components and the color component dependencies are captured via the non-diagonal entries in the covariance matrices of the multivariate Gaussian densities. There are two different methods of grouping the 1 2 three corresponding hidden states S XB;k;i;j ; S XB;k;i;j ; X3 S B;k;i;j to form the hidden state of the MGMM for ~ wB;k;i;j . The first uses each combination of the three hidden states to represent one hidden state of wavelet vector ~ wB;k;i;j (Fig. 2a). If the number of the hidden states for one wavelet coefficient is M, then there will be M · M · M states for each wavelet vector. In the second method, the three
X1
X2 (a)
X3
X1
X2 (b)
X3
Fig. 2. Multivariate Gaussian mixture model, where the black nodes represent the hidden state random variable and the white nodes represent the color component wavelet coefficient. (a) Each color component value is associated with different hidden state random variables. (b) All three color component values share the same hidden state random variable.
1714
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
3 1 2 hidden states S XB;k;i;j ; S XB;k;i;j ; S XB;k;i;j are ‘‘tied up’’ to have a same hidden state (Fig. 2b). If the number of the hidden states for one wavelet coefficient is M, there will still be M states for each wavelet vector. For example, if we assume two hidden states (M = 2) for each coefficient, i.e., 0 and 1, the wavelet vector in the second grouping method will have two hidden states: 000 and 111. On the other hand, the first method will produce eight hidden states for the wavelet vector: 000, 001, 010, 011, 100, 101, 110, and 111. The first method characterizes the distribution density of the wavelet coefficient vectors more completely than the other method. However, the second method tends to be more practical, since it simplifies computation and represents color texture effectively. So we use the second grouping technique in our study. Here we also assume M = 2 and zero means for the GMM of one coefficient. Thus, the marginal distributions of the wavelet coefficient vector ~ wBj;k;i in the j scale can be expressed as
~ wBj;k;i fjB ð~ xÞ ¼
M 1 X
pmj gð~ x; C mj Þ;
ð3Þ
m¼0
where gð~ x; CÞ denotes the zero-mean multivariate Gaussian density with covariance matrix C, i.e. 1 gð~ x; CÞ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi expð~ xÞ; xT C 1~ 2 ð2pÞ j detðCÞj
ð4Þ
where n is the number of color components or the dimension of the color space. In order to capture the persistence in a subband, the color wavelet coefficient vectors are organized into a quad-tree structure that connects each vector to its four child vectors at the next intermediate scale of the same location (Fig. 3). Persistence is quantified the same way as the WD HMT for the gray level image. There are also state probability transition matrices to describe the parent ! child link relationships of these vectors. Thus, tied within scales, the HMT for color texture is parameterized as follows: hHMT
color
¼ fpBj ðmÞ; eBj;j1 ðm; nÞ; C B;j;m jB 2 fHH ; LH ; HLg; j ¼ 1; 2; . . . ; J ;m; n ¼ 0; 1g;
Fig. 3. Hidden Markov tree model for color texture, where the black nodes represent the hidden state and the three white nodes in the eclipse represent a three-dimensional color vector. The three color components are only determined by one hidden state random variable, where the link between the black nodes captures the interaction of the different scales.
where eBj;j1 ðm; nÞ is the transition probability of the Markov chain from scale j to scale j 1 in sub-band B. P BJ represents the probability mass function of the hidden state on the J scale. Given the wavelet transform of a color image ~ , the hHMT_color can be estimated by employW ing the tree-structured EM training algorithm, which maximizes the HMT model likelihood ~ jhHMT color Þ. The difference from the EM trainf ðW ing algorithm proposed by Crouse et al. (1998) is that the GMM in the HMT is replaced by the MGMM. Besides, the model posterior likelihood ~ j hHMT color Þ is defined as f ðW ~ jhHMT color Þ f ðW ¼
X
NX J 1
B2fHH ;LH ;HLg k;i¼0
log
1 X
! fm ðT BJ;k;i jhHMT color ;mÞ
;
m¼0
ð5Þ
where T BJ;k;i denotes the wavelet sub-tree rooted at ~ wJ ;k;i and (k, i) indexes the spatial location of wave~ j hHMT color Þ can be let vector in J scale. The f ðW computed using an upward–downward algorithm. In our study, we apply our new WD HMT for the color image to color texture analysis, and obtain a superior result.
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
4. Color texture classification Color texture classification is one of the most important applications of color texture analysis, and it involves the identification of the color texture class given a homogeneous color texture region. Limited classification methods use the combined information of color and texture, and even when they do, most of them deal with the two aspects using separate methods. In our work, we adopt and evaluate four different approaches combining both color and texture aspects in one model or method, including wavelet energy correlation signature and three different wavelet-domain hidden Markov models. In this section, these wavelet-based methods are introduced, together with their respective classification methods. 4.1. Wavelet energy correlation signature Van de Wouwer et al. (1998) has defined C XB;jm ;X n ¼
Nj X Nj X 1 wX m wX n ; N j N j k¼0 i¼0 B;j;i;k B;j;i;k
m wXB;j;i;k
is the wavelet coefficient at (i, k) where location at j scale in B (B 2 {HH, LH, HL}) subband and Xm is the color plane (m = 1, 2, 3). The set fC XB;jm ;X n jm; n ¼ 1; 2; 3;m < n;B 2 fHH ; HL; LH g; j ¼ 1; 2; . . . ; J g is called the wavelet energy correlation signature. They capture the energy distribution of the wavelet coefficients over the scale, sub-band and color plane for m = n, and the others (m 5 n) represent the covariance between different color planes. The vector of the wavelet energy correlation signature (~ 1 ¼ fC XB;jm ;X n jm; n ¼ 1; 2; 3;m < n;B 2 fHH ; HL; LH g;j ¼ 1; 2; . . . ; J g) has proven to be efficient for color texture characterization (Van de Wouwer et al., 1998). In our work, we carry out a 5-scale discrete wavelet transform (DWT) on each color plane separately. Thus, the size of the feature vector is 135 = 9 · 5 · 3. A k-nearest neighbor (KNN) classifier is used to estimate the classification performance using the wavelet energy correlation signature. Given the vectors (~ 1) of the training images and that of the test image, the class of the test image is determined by the dominant class of the k-nearest
1715
neighbor training images based on a distance function. Here we adopt the Euclidean distance measure for simplicity. 4.2. WD HMM methods The feature extraction step of the WD HMM methods becomes a maximum likelihood (ML) estimator for model parameters of image data. Then given the estimated model parameter ~ , we can hHMT_color and the transformed image W ~ compute the posterior likelihood f ðW jhHMT color Þ using the closed formulas (3)–(5). Thus, in the case in which the prior probabilities to the texture classes are equal and where the goal is to minimize the overall probability of classification error, the optimal decision becomes the maximum likelihood (ML) rule, which is to choose the class making the observed data most likely. Given the WD HMM models (fhcHMT color jc ¼ 1; 2; . . . ; N c g) estimated by the training images and the transformed ~ , the ML classifier can be image of the test image W developed as ~ jhc C ML ¼ arg max f ðW HMT c2f1;...;N c g
color Þ.
ð6Þ
All of the wavelet-domain statistical models studied here share the same classifying framework described as in (6). To obtain the wavelet coefficients for each color component, we perform 5-scale DWT on each color plane of the color image separately. In addition to the method proposed in Section 3, we have chosen two other wavelet-domain hidden Markov models to make comparisons with our method. The first applies the WD HMT for the gray level image to the luminance image of the original color image in order to show the texture classification performance without considering the color information. The second method models each of the three color planes independently using the WD HMT for the gray image, where the different color planes are assumed to be independent of each other. The color representation often determines the performance of the color image analysis approach. In addition to the RGB color space, we have also performed the wavelet-domain hidden Markov
1716
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
models on the I1I2I3 color coordinate, which transforms an image to an orthogonal basis in which the axes are statistically un-correlated. The transformation from the RGB color space to the I1I2I3 space can be written as (Ohta, 1985): 2 3 2 32 3 I1 0.333 0.333 0.333 R 6 7 6 76 7 4 I2 5 ¼ 4 0.500 0.000 0.500 54 G 5. I3
0.500
1.000
0.500
B
The reason why we use this color space depends on the fact that the K–L transform tends to de-correlate the information present in three color components. So, by transforming the RGB image to the I1I2I3 space, we can treat the three color planes as independent ones without much loss of information.
5. Experiments and discussions Our experiments are performed on the test suite Qutex_TC_00013 provided by Qutex texture database. 68 real-world RGB (746 · 538) color images from natural scenes are selected: canvas001, canvas002, canvas003, canvas004, canvas005, canvas006, canvas009, canvas011, canvas020, canvas021, canvas022, canvas023, canvas024, canvas025, canvas026, cardboard001, carpet001, carpet004, carpet005, carpet006, carpet008, carpet009, carpet010, carpet011, carpet012, foam002, foam003, paper001, rubber001, tile001, tile002, tile003, tile004, tile005, tile006, tile007, granite001, granite002, granite003, granite004, granite005, granite006, granite007, granite008, granite009, granite010, sandpaper001, sandpaper002, sandpaper003, sandpaper004, sandpaper005, sandpaper006, sandpaper007, sandpaper008, wool001, wool002, wood001, barleyrice001, barleyrice002, barleyrice003, barleyrice004, barleyrice005, barleyrice006, barleyrice007, barleyrice008, barleyrice009, barleyrice010 and barleyrice011. A database of 1360 color texture images (128 · 128) of 68 texture classes was constructed by splitting each texture (746 · 538) into 20 non-overlapping subimages (128 · 128). Half of the sub-textures in that database are used as the training set, while the rest
of the 680 sub-images serve as the test set. We carefully follow the training and testing protocol with Qutex_TC_00013. The following six classification experiments have been conducted: (1) The intensity images are generated by computing the luminance of the color images, hereby discarding the color information. A 5-scale Daubechies-8 wavelet transform is performed on the intensity images and then the gray WD HMT parameters of each texture class are estimated using the 10 transformed training intensity images for each class. Finally, we use the maximum likelihood (ML) ruler to classify the test images according to the 68 HMT models. (2) Each color component (RGB) is wavelet transformed (5-scale db8) separately and then the wavelet energy correlation signature vectors are computed for each detailed subimages, including training and test images. Here we adopt a KNN (k = 7) classifier. (3) Each color component (RGB) is also wavelet transformed (5-scale db8) separately and then the gray WD HMT model for each color plane of each class is trained separately using the training images. Finally, we use the maximum likelihood (ML) ruler to classify the test images. (4) Each color component (RGB) is also wavelet transformed (5-scale db8) separately and then our proposed model of each class is estimated using the training images. The classification of the test images is also based on the maximum likelihood (ML) classifier. (5) We conduct the same experiment as in experiment 3, except that the RGB space is replaced by the I1I2I3 space. (6) We conduct the same experiment as in experiment 4, except that the RGB space is replaced by the I1I2I3 space. We will discuss the selection of hyper parameters in the experiments, including the depth of wavelet transform, the wavelet basis, and the neighboring number of KNN classifier used in experiment 2. There are two factors for us to select the depth of wavelet transform. First, the depth
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
1717
Table 1 Classification result (PCC) of six methods on 68 color texture images Texture
Method 1 PCC
Method 2 PCC
Method 3 PCC
Method 4 PCC
Method 5 PCC
Sample1 Sample2 Sample3 Sample4 Sample5 Sample6 Sample7 Sample8 Sample9 Sample10 Sample11 Sample12 Sample13 Sample14 Sample15 Sample16 Sample17 Sample18 Sample19 Sample20 Sample21 Sample22 Sample23 Sample24 Sample25 Sample26 Sample27 Sample28 Sample29 Sample30 Sample31 Sample32 Sample33 Sample34 Sample35 Sample36 Sample37 Sample38 Sample39 Sample40 Sample41 Sample42 Sample43 Sample44 Sample45 Sample46 Sample47 Sample48 Sample49 Sample50 Sample51 Sample52 Sample53
1 0.5 0.5 0.9 0.7 0.9 1 1 0.4 0.8 0.8 0.5 0.5 0.5 0.9 0.4 0.3 1 0.4 0.1 0.6 0.2 0.1 0.8 0.2 0.9 0.6 0.7 0.7 0.4 0.4 0.4 0.4 0.1 0.2 0.1 0 0.9 0 0.4 0.3 0 0.3 0.3 0.5 0.4 0.2 0.2 0 0.4 0 0.5 0.7
1 1 1 1 1 1 1 1 1 1 1 1 0.8 1 1 1 1 0.9 0.9 0.7 0.7 0.9 0.6 0.8 0.7 0.5 0.8 1 0.9 0.8 0.6 0.7 0.8 0.9 0.7 0.5 0.2 1 0.9 0.7 0.9 0.7 0.6 1 0.9 1 0.7 0.5 0.5 0.8 0.6 0.8 1
1 1 1 1 1 1 1 1 1 1 1 0.8 1 0.9 1 1 1 1 0.8 0.9 0.8 0.9 1 1 1 0.9 1 1 1 0.9 0.6 0.8 0.9 0.9 0.6 0.8 0.7 1 0.6 0.8 0.9 0.5 0.7 0.7 0.4 0.8 0.9 0 0.2 0.4 0 0.8 0.9
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0.9 1 1 1 1 1 0.9 1 1 1 1 0.8 1 1 0.9 0.8 0.7 0.7 1 0.9 0.9 0.9 0.6 0.7 1 0.7 1 0.4 0.9 0.8 0.7 0.6 0.9 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0.9 0.8 0.8 0.9 1 0.9 0.9 1 1 1 1 0.9 1 1 0.6 0.6 0.7 0.7 1 0 0.9 0.9 0.5 0.7 1 0.8 1 0.7 0.1 0.1 0.7 0.6 0.4 1
Method 6 PCC 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0.7 0.7 1 1 0.8 0.2 1 1 0.6 1 1 0.7 1 0.9 0.5 0.2 0.9 0.7 0.6 0.9 (continued on next page)
1718
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
Table 1 (continued) Texture
Method 1 PCC
Method 2 PCC
Method 3 PCC
Method 4 PCC
Method 5 PCC
Method 6 PCC
Sample54 Sample55 Sample56 Sample57 Sample58 Sample59 Sample60 Sample61 Sample62 Sample63 Sample64 Sample65 Sample66 Sample67 Sample68 Mean of PCCs Standard deviation of PCCs
0.6 0.6 0.3 0.8 0.3 0.1 0.1 0 0.1 0.2 0 0 0.3 0.9 0.2 0.4338 0.3079
1 1 1 0.9 0.3 0.2 0.3 0.2 0.3 0.4 0.7 0.3 1 0.2 0.3 0.7662 0.2584
1 1 1 1 0.3 0 0.3 0.1 0.6 0.4 0.6 0.5 0.8 0.1 0.5 0.7647 0.2951
1 1 1 1 0.3 0.6 0.4 0.3 0.3 0.4 0.5 0.2 1 0.5 0.7 0.8515 0.2256
1 1 0.9 1 0.3 0 0.4 0.2 0.5 0.7 0.5 0.2 1 0.6 0.4 0.7912 0.2900
1 1 1 1 0.3 0.5 0.4 0.3 0.7 0.6 0.6 0.3 1 0.5 0.2 0.8500 0.2483
must be large enough to make most of the textures in our database homogeneous in this small size 2d · 2d patch, where d represents the depth. Second, the depth should be selected as small as possible to avoid the overlearning problem and reduce the computation complexity. We have made a tradeoff between these two factors and selected the depth as 5. Anyway, to select the best depth is an important issue, which will be studied in the future. Focusing on the comparisons between different models, we have not considered the selection of the optimal wavelet basis for texture characterization and just selected the db8 wavelet for all the wavelet-based methods we adopt. For the neighboring number of the KNN classifier, we have selected the number 7 among all the odd numbers from 1 to 21, since 7 provides the best average percentage of correct classification over the 68 textures. Here we use the average percentage of correct classification (PCC) to evaluate the classification performance for each method, as shown in Table 1. Compared with method 1 (43.38% 0.3079), 2 (76.62% 0.2584), 3 (76.47% 0.2951) and 5 (79.12% 0.2900), methods 6 (85% 0.2483) and 4 (85.15% 0.2256) provide a better PCC over the 68 color textures, and an overall better numerical stability. Given the average PCCs on the test set in Table 1, we have performed statistical signifi-
cance tests to show that methods 3, 4, 5 and 6 outperform method 1 with a probability above 99%. Method 1 fails to consider the color information, while the later five methods gain their improvement by considering color and structure information together. This proves that we can obtain a more effective characterization of color texture by combining color and structure information. Moreover, methods 4 and 6 outperform the energy signature method at a confidence level above 99%. The reason for this is that the hidden Markov models can represent the statistics of wavelet coefficients more precisely than the energy signature method, which only captures the mean and covariance matrix of the wavelet vectors. We also observe that method 4 outperforms method 3 at a confidence level above 99% and method 6 also outperforms method 5 at a confidence level above 99%, because methods 3 and 5 ignore the correlated information of different color components. However, method 4 gains more improvement over method 3 (0.8515 0.7647 = 0.0868) than method 6 over method 5 (0.8500 0.7912 = 0.0588). This situation can be explained by the statistical un-correlation of different color components in the I1I2I3 color space and thus the dependencies between different color planes are weaker than those in the RGB color space.
Q. Xu et al. / Pattern Recognition Letters 26 (2005) 1710–1719
Therefore, ignoring the correlations between color planes in the I1I2I3 space causes less information loss than doing so in the RGB space.
6. Conclusions In this paper, we have extended the wavelet-domain hidden Markov model method to color texture analysis. Among the four wavelet-based methods we investigated, our proposed HMT for color texture provides the best classification results (over 85% average PCC), because it captures the dependencies between both the scales and the interactions of the three color components. In addition, this approach has better numerical stability than other methods. However, we have ignored the dependences across the three orientations and the high computational complexity is still a big disadvantage. This would make our model inappropriate for the image retrieval systems. In the future, we intend to simplify the computation by investigating faster learning methods, and at the same time, modify our approach for capturing the interactions of the different orientations and color planes together.
1719
Markov models. In: Proc. 34th Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, October. Fan, G., Xia, X.G., 2000b. Multi-scale texture segmentation using hybrid contextual labeling tree. In: Proc. IEEE Internat. Conf. on Image Processing, Vancouver, Canada. Krishnamachari, S., Chellappa, R., 1997. Multi-resolution Gauss–Markov random field models for texture segmentation. IEEE Trans. Image Process. 6 (2). Laine, A., Fan, J., 1993. Texture classification by wavelet packet signatures. IEEE Trans. Pattern Anal. Machine Intell. 15 (11), 1186–1191. Ohta, Y., 1985. Knowledge Based Interpretation of Outdoor Natural Scenes. Pitman Publishing, London. Romberg, J.K., Choi, H., Baraniuk, R.G., 2001. Bayesian treestructured image modeling using wavelet-domain hidden Markov models. IEEE Trans. Image Process. 10, 1056– 1068. Tan, S.C., Kittler, J., 1993. Color texture classification using features from color histogram. In: Proc. 8th Scand. Conf. on Image Analysis, pp. 807–813. Tardif, P.M., Zaccarin, A., 1997. Multi-scale autoregressive image representation for texture segmentation. Image Process. VIII 3 (026), 327–337. University of Oulu texture database. Available from:
. Van de Wouwer, G., Livens, S., Scheunders, P., Van Dyck, D., 1998. Color texture classification by wavelet energy correlation signatures. Pattern Recognition, Special issue on Color and Texture Analysis. Wan, X., Kuo, C.C.J., 1996. Color distribution analysis and quantization for image retrieval. In: SPIE Proc., vol. 2670.
References Further reading Caeli, T., Reye, D., 1993. On the classification of the image regions by color, texture and shape. Pattern Recognition 26 (4), 461–470. Choi, H., Baraniuk, R., 2001. Multi-scale image segmentation using wavelet-domain hidden Markov models. IEEE Trans. Image Process. 10 (9), 1309–1321. Crouse, M.S., Nowak, R.D., Baraniuk, R.G., 1998. Waveletbased statistical signal processing using hidden Markov model. IEEE Trans. Signal Process. 46 (4), 886–902. Do, M.N., Vetterli, M., 2002a. Rotation invariant texture classification and retrieval using steerable wavelet-domain hidden Markov models. IEEE Trans. Multimedia 4 (4). Do, M.N., Vetterli, M., 2002b. Wavelet-based texture retrieval using generalized Gaussian density and Kullback–Leibler distance. IEEE Trans. Image Process. 11, 146–158. Fan, G., Xia, X.G., 2000a. Maximum likelihood texture analysis and classification using wavelet-domain hidden
Crouse, M.S., Baraniuk, R.G., 1997. Contextual hidden Markov models for wavelet domain signal processing. In: Proc. 31st Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, November. Fan, G., Xia, X.G., 2001a. Image de-noising using local contextual hidden Markov model in the wavelet-domain. IEEE Signal Process. Lett. 8, 125–128. Fan, G., Xia, X.G., 2001b. Improved hidden Markov models in the wavelet-domain. IEEE Trans. Signal Process. 49, 115– 120. Fan, G., Xia, X.G., 2001c. A joint multi-context and multiscale approach to Bayesian image segmentation. IEEE Trans. Geosci. Remote Sensing 39 (12), 2680–2688. Fan, G., Xia, X.G., 2003. Wavelet based texture analysis and synthesis using hidden markov models. IEEE Trans. Circ. Syst.—Part I 50 (1), 106–120.
Pattern Recognition Letters 26 (2005) 1609–1619 www.elsevier.com/locate/patrec
Detected edge position evaluation using measured acquisition system parameters R.C. Staunton
*
School of Engineering, University of Warwick, Gibbit Hill Road, Coventry CV4 7AL, UK Received 4 February 2004; received in revised form 14 December 2004 Available online 24 February 2005 Communicated by E. Backer
Abstract A method of determining edge detector performance from tests on synthetic images is presented. The position of a step edge is defined and its shape modified by the acquisition systemÕs measured point spread function. These images contain both accurate positional information and realistic edge profiles. Results for a Canny detector have been presented and its optimum operating parameters determined for the given acquisition system. 2005 Elsevier B.V. All rights reserved. Keywords: Edge detection; Edge positional accuracy; Image acquisition system; Point spread function; Canny detector
1. Introduction An edge can be defined as an abrupt brightness change between one region in an image and another. Edge detectors can be used to find these boundaries. In the past several methods have been devised to compare the performance of edge detectors. To measure performance, it is necessary to establish a ground truth to define the actual edge position to within sub-pixel accuracy. The detec-
*
Tel.: +44 2476523980; fax: +44 2476418922. E-mail address: [email protected]
torÕs position estimate can then be compared with this. However, generally defining the ground truth is difficult. Synthetic edge images have been widely used. They are convenient in that the edges can be constructed from a precisely known ground truth, Lyvers and Mitchell (1988). However, synthesis may not result in realistic edge images when simple, general models of image blur are used. An alternative is for a panel of experts to define the ground truth of an actual, captured image. Bowyer et al. (2001) have done this for a set of 60 images that were captured by a variety of acquisition systems, however it is unlikely that the observers will be able to estimate the ground truth to sub-pixel
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.007
1610
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619 Discrete CCD Array - Discrete Continuous
Digitizer
Part Discrete Camera
A/D
Frame Memory
2D MTF
Fig. 1. TV camera and digitizer system.
accuracy and the procedure requires the participation of several experts. Neither approach can easily determine how a particular detector will work with images captured by a specific acquisition system. To overcome the limitations of the previous methods, the tests reported here use images synthesized from ground truth step edges to ensure high positional accuracy, which are then convolved with the previously measured point spread function (PSF) of the acquisition system to give realistic edge profiles. To enable this, the PSF must be measured to sub-pixel accuracy. Acquisition systems are comprised of several components that blur, that is low-pass filter the image of an otherwise ‘‘sharp’’ scene. The lens and CCD sensor elements are 2D filters, whereas the camera and digitizer electronics are 1D filters. Fig. 1 shows a typical arrangement of acquisition system components. The image is continuous when it is focused on to the cameraÕs CCD array, after that it is discrete. The cameraÕs electronics may, or may not, partly reconstruct a continuous signal in the form of a raster-scanned image, but within the computerÕs frame grabber, the Analogue to Digital converter will re-sample and digitize it. The Canny detector, Canny (1986) is widely used and so has been utilized here to illustrate the measurement capability of the new technique. It has three parameters: Sigma (r) that sets the smoothing filter cutoff, and a high and a low threshold, which adjust the hysteresis operation to determine if a pixel is part of an edge. In the experiments reported in this letter, characteristic curves were compiled, for an acquisition system, of the true and false edge-pixel detection perfor-
mance vs. each detector parameter. The letter is organized as follows. First previous edge synthesis techniques and how they have lead to the proposed method are described in Section 2, together with an outline of the method used to measure the PSF of the acquisition system. Section 3 describes the formation of sets of test edges. Section 4 describes the methods used to test the detector and provide quantitative results. The results together with a discussion of how they can be used for detector parameter selection follow in Section 5. Finally the main conclusions are in Section 6.
2. Edge synthesis Edge models have been devised and listed by Davis (1975). He identified the sudden step, slanted step, roof, and line cases as the most likely to occur. The sudden step, or scaled and translated Ôunit step functionÕ has been identified as the most difficult to detect with positional accuracy, Davis (1975). However the slanted step has long been realized as a more realistic model of a real edge, Davies (1984) and Lyvers and Mitchell (1988). The slanted step results from the blurring of sudden step edges in the scene. The slanted step was modeled by Davis (1975) as a connection of piecewise line components, but the actual blurred step is ÔSÕ shaped, Staunton (1998), and appears as in Fig. 2 as the smoothed edge. In practice this edge will also be sampled and noisy. In this work a sudden step, the ground truth of which is easy to define, is smoothed by convolution with the acquisition systemÕs PSF to produce a smoothed edge on which to test the detector.
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
1611
250
Brightness
200
150
100
50
0 –15
–10
–5
0
5
10
15
Pixel number
Fig. 2. Step and smoothed edges.
Edge synthesis begins with an accurately positioned edge, usually a step edge (Lyvers and Mitchell, 1988). It removes the requirement for a panel of experts to estimate the ground truth, but the edge must be allowed to pass through any sub-pixel point within a pixel, and the brightness profile across the edge must realistically match that in a real image captured by an acquisition system if a reliable test of the detector is to be performed. Fig. 2 shows a step edge and a smoothed edge that results after capture by an acquisition system comprising a lens that blurs the image, CCD elements that average the image over a finite area, and camera electronics that low pass filter the image signal before digitization. Lyvers and Mitchell (1988) have compared several detectors on a set of synthetic edges. They used a theoretical model of step edge smoothing by a square mosaic of CCD elements. They set a step edge to pass through a point within the central element, and then by calculating the element area on either side of the edge they found, by proportion, a sample value for each element in the CCD array. Fig. 3 illustrates the arrangement for a 3 · 3 array. The step edge passes through the central element. Towards the lower left corner the image has a gray level brightness of ÔaÕ, and across the edge the brightness is Ôa + hÕ. The normal to the edge is at h degrees to the horizontal, and the perpendicular distance from central point x to the edge is l. The sampled brightness for the
Fig. 3. Ideal step edge passing through an ideal CCD element. Sampling point x, edge translated l from x and with orientation h.
central pixel is then calculated as the average brightness over that pixel. A discrete synthetic image containing a smoothed edge was thus formed with some pixels, brightness a, some brightness a + h, and edge pixels with intermediate brightness. Lyvers and Mitchell (1988) tested detectors on such modeled edges for both angle and gradient measurement accuracy. Many such edges at various orientation and translation from the central image point were used in the testing of each detector to enable performance statistics to be compiled. This technique models a single CCD element realistically as within the element, charge is averaged across the area of the element. Electronics convert the charge to a brightness ‘‘sample’’ value for that element. Lyvers and Mitchell (1988) method has the advantages of a well defined ground truth and easily compiled measurement accuracy statistics, but filtering by the lens and the acquisition systemÕs electronics were not modeled. White and Schowengerdt (1994) again used a step edge model, but which had been convolved with a theoretically determined PSF. Their system PSF was represented by the cascaded PSFs of the lens and CCD array components. The effect of the systemÕs electronic filtering was not considered. The continuous image g(x, y) was thus modeled by
1612
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
gðx; yÞ ¼ f ðx; yÞ hop ðx; yÞ hccd ðx; yÞ
ð1Þ
where f(x, y) was the step edge object, hop(x, y) the PSF of the optics, hccd(x, y) the PSF of the CCD array, and * represents convolution. The image g(x, y) was then sampled to produce a discrete test image. Pillbox and Gaussian PSF models of the optics were implemented together with a rectangular PSF model of the CCD array. The effect of varying the width of the overall PSF on the detector was measured and found to significantly affect the accuracy of the detector. The theoretical analysis by White and Schowengerdt (1994) has shown that blurring by system components is an important factor in edge detector design. In the work reported here, the measured PSF of the system was used in the test edge synthesis. For accurate synthesis the Fourier transform of the PSF must be calculated for frequencies beyond the Nyquist limit of the CCD array. The average PSF of the system is required for detector testing. In general the PSF is spatially variant, but unless a different detector will be used at each pixel in the image, the average PSF is sufficient. Knife-edge techniques have been used for such measurements and the large data sets involved exploited to give low noise PSFs with sub-pixel accuracy. Early implementations of the technique, Reichenbach et al. (1991), were limited to measurements in the vertical and horizontal directions and required several parallel edges. The use of spatial domain calculations, Tzannes and Mooney (1995) has enabled a single knife-edge to be used, and the consideration of plane waves and interpolation has enabled full 2D measurements to be made with sub-pixel accuracy, Staunton (1998). Such measured PSFs comprise the cascaded PSFs of all the acquisition systemÕs components including the lens, the CCD array, and the electronics. For the systems measured, the PSF of the lens was found to dominate the 2D PSF resulting in circularly symmetric shapes. However in several cases, low pass filtering in the electronics was observed to limit the PSF in the horizontal image direction. As expected, the PSF width was found to be a function of the lens aperture diameter and lens defocus.
The PSF measurement technique, as used in the work reported here, is described in Staunton (1998). The knife-edge was constructed by placing a straight edged metal strip on top of a DC light box. The knife-edge was imaged so that approximately half the image was bright and half dark. Considering a simple model of the acquisition system, within the image, the knife-edge cuts each pixel along its length so that a different light/dark ratio covers the integrating area of each pixel. In practice other low pass filters in the system spread the edge over several pixel around its nominal position. Starting our discussion with an almost horizontal edge, that is the edge is not at exactly 0, then the position of the edge within the pixel and the brightness value sampled change as we step between pixels along the edge. Now 1D profiles across the image of the edge can be analyzed, to give a series of discrete, low resolution edge spread functions (ESFs), one for each pixel along the knife-edge. Many such ESFs can be measured (e.g. 512 for a 512 · 512 image). Now, as the edge is not at exactly 0, few of these low resolution ESFs will be identically sampled. At the next stage, the sampled values in the many low resolution ESFs are aligned using a spline fitting technique and assembled to form a single high resolution ESF. An example high resolution ESF for a particular system is shown in Fig. 2. Here, the high resolution ESF was resampled on a 1/10 pixel grid and averaging of the many low resolution ESFs has resulted in a high signal to noise ratio (SNR). With StauntonÕs (1998) method it is possible to measure the high resolution ESF for any knifeedge angle. This was achieved by calculating the low resolution ESF samples using an interpolation between known points on a line parallel to the edge. For the results presented here, the high resolution ESFs were obtained with the knife-edge oriented in 15 steps from the horizontal to the vertical to investigate differences resulting from non-square pixel shapes and 1D filters within the acquisition system. The systemÕs PSF for a particular direction was obtained by differentiating the ESF. These were found to be almost noise free because the initial high resolution ESFs were virtually noise free. The PSFs for each measurement angle were then transformed to the Fourier
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
domain and the absolute values taken to form the systemÕs modulation transfer function (MTF) for each measurement angle. The set of 1D MTFs were assembled into a 2D plot to enable easy viewing of the systemÕs low-pass filtering characteristics as shown in Fig. 4. For clarity, only one quadrant has been plotted, the remaining quadrants were symmetric with this. It is easier to assess the lowpass filter band-limiting on a polar plot of the MTF as shown in Fig. 5. The plots are for the half power (3 dB) slice through the plot in Fig. 4. Two different system plots have been shown. System (b) has almost circular band limiting. A poor quality or out of focus lens may result in such band limiting. The cut-off frequency is at 0.4 of the Nyquist frequency. System (a) has a vertical cut-off frequency of 0.54 and a horizontal cut-off of 0.26. The lower horizontal cut-off results from electronic low-pass filtering in the camera or anti-alias filtering in the frame grabber. Such filtering can only be applied, in real time, in the horizontal direction. System (a) was chosen for the detector tests and from Fig. 5 the highest cut-off frequency in the MTF was for the vertical knifeedge profile. The corresponding PSF was therefore chosen to represent the system because convolving this with a step edge will produce the sharpest (most step like) high resolution edge for detector testing, and the sharpest edge is considered to be the most difficult to accurately detect, Lyvers and Mitchell (1988).
Normalised MTF
1.4 1.2 1 0.8 0.6 0.4 0.2 0 25 Ho 20 rizo nta 15 l Fr 10 equ 5 enc y (I nde 0 0 xn o.)
5 al ertic
10
25 20 15 .) o n x ( Inde enc y
u Freq
V
Fig. 4. Quadrant of a 2D MTF.
0.8
90
1613
Normalised Frequency, Angle (degrees) 60
0.7 0.6 0.5
30
0.4 0.3 0.2 (b) (a)
0.1 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0 0.8
Fig. 5. Polar plots of the 3 dB modulation points of the 2D MTFs obtained from edge normals at angles of 0 to 90 to the horizontal: (a) a system limited by camera electronics and (b) a system limited by lens defocus.
3. The set of test edges used For the edge synthesis used here, a precisely positioned high-resolution step edge was convolved with the high-resolution PSF (HR-PSF) measured using the method reported in Section 2. A realistic edge image was thus formed with a precisely defined ground truth. This edge was then re-sampled (decimation by 10) at the correct resolution to form images on which to test the detector. The acquisition system that was modeled comprised a TV camera with a CCD array of a 512 · 512 pixels, a 16 mm fixed focus lens, and a frame grabber board. The objective was to synthesise a large set of realistic, re-sampled test edge images with orientations from 0 to 90 each with a ground truth positioned to within 1/10th pixel resolution. This angle range was sufficient as the Canny detector is symmetric about both its horizontal and vertical axes. Each edge in the set was randomly positioned to sub-pixel accuracy within the central pixel of its image, resulting in different sample values for each individual edge image in the set of test images. In
1614
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
practice, to achieve the objective each test edge image was paired with a high-resolution ground truth (HR-GT) image so that the eventual edge detector output could easily be compared with the actual edge position for every pixel in the image. The high-resolution edge image was resampled 5000 times by a low-resolution grid of 23 · 23 points at random orientation and translation to obtain the set of low-resolution, grey-scale test images to be used for detector testing. Finally zero mean, Gaussian distributed random noise was added to the brightness of each pixel of each edge image. The signal to noise ratio was defined as SNR ¼ 20log10 ðc=rn Þ dB;
ð2Þ
where rn was the standard deviation of the noise, and c was the height of the step edge. For the re-
Fig. 6. A high-resolution edge image overlaid with a translated 3 · 3 sampling point grid and a vertical ground truth edge.
0 0 1m 9 0 0 0 0 0
0 0 0 11 0 0 0 0 0
0 0 0 11 0 0 0 0 0
0 0 0 0 10 0 0 0 0
0 0 0 0 11 0 0 0 0 (a)
0 0 0 0 10 0 0 0 0
0 0 0 0 0 11 0 0 0
0 0 0 0 0 11 0 0 0
0 0 0 0 0 9 1 0 0
sults reported here c was set to 200 brightness units in a 256-unit resolution system. A set of test images was formed for each SNR used. Fig. 6 shows a section of a high-resolution step edge image after convolution with the HR-PSF. The smooth change in brightness from the top to the bottom of the edge with horizontal distance can bee seen. The low-resolution resampling is illustrated for a simplified 3 · 3 case. For this resampling example, a translation, but no rotation has been applied to the grid of sampling points. The brightness values at the position of each of the sub-pixel grid points are selected as the set of resampled values. The ground truth position of the step edge has been overlaid on the image as a vertical white line. A low-resolution ground truth image (LR-GT) was formed for each low-resolution test image by investigating a HR-GT image with the same random orientation and translation as the test image. A list was then generated of each sub-pixel in the HR-GT image that the detected edge should pass through. This list was then processed to produce a LR-GT image with the same resolution as the low-resolution test image. Each pixel in the LRGT contained the sum of the HR-GT edge subpixels within its boundary. Fig. 7(a) shows the central part of a LR-GT image. The steps in the generation of the low-resolution test images were: (1) Obtain a high-resolution edge image by convolving a step edge with the systemÕs PSF. (2) Resample the image at (1) using a low-resolution grid, where the grid has been positioned and rotated randomly in the image. (3) Add noise to each resampled value.
0 0 0 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0 1 1f 0 0 0 0
0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 (b)
0 0 0 0 1 1f 0 0 0
0 0 0 0 0 1 0 0 0
Fig. 7. For a single test: (a) accumulated LR-GT and (b) detected edge map.
0 0 0 0 0 1 0 0 0
0 0 0 0 0 1 1 0 0
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
(4) Investigate the HR-GT image with the random orientation of the edge considered at (2), and noting the randomly generated position variables from (1), generate a list of each sub-pixel in the HR-GT image that the detected edge should pass through. Assemble a LR-GT image. (5) Repeat steps (2)–(4) 5000 times.
4. Detector testing The Canny detector implementation required five steps: convolution of the image with a Gaussian filter (for a particular r); computation of the first derivatives; computation of the edge magnitude; non-maximal suppression; hysteresis (for particular upper and lower-thresholds). In normal use, the r and threshold parameters are adjusted experimentally to optimize the detectorÕs operation, but here, one parameter was varied while the other two were kept constant. For a range of SNRs, sets of 5000 test images were applied sequentially to the detector for each chosen parameter setting. To avoid boundary effects, in that the filter used here in the detector required a support of up to 14 · 14 pixels, only the central 9 · 9 pixels of each 23 · 23 pixel image were fully edge detected. Statistically this small central area was sufficient as 5000 such images were processed. The resulting edge maps were then compared with the appropriately oriented and translated ground truth image. Statistics were compiled for the number of true positive (TP) and false positive (FP) edge points. This was done for TP by accumulating the sub-pixel edge points from the LR-GT image wherever a measured edge pixel was identified in the edge map. For FP the number of detected edge points for which there were no LR-GT subpixels were accumulated. The process is illustrated in Fig. 7 where the 9 · 9, central, processed pixels of a LR-GT and a detected edge map pair are shown. Here the LR-GT image contains M = 81 pixels, in total there are N = 95 sub-pixel edge points, and the edge map contains 12 detected edge pixels indicated by binary Ô1sÕ. Examining the figure we count TP = 94, FP = 2, and missed points
1615
(N-TP) = 1. The missed point is subscripted m, and the false points, f, in the figure. For this example the percentage of TPs, TP% = TP · 100/ N = 98.9% and the percentage of FPs, FP% = FP · 100/M = 2.5%. The steps in the detector testing procedure for each image were: (1) Edge detect the central 9 · 9 pixels of the low resolution image. (2) Compare the edge map with the LR-GT image. (3) Accumulate TP and FP statistics.
5. Results The following results are for SNRs of 10–40 dB, where 10 dB represents a very low, 20 dB a low, and 30–40 dB a high SNR image. For SNR = 10 dB we can expect 95.4% of the noise samples to be in the range ±126, for SNR = 20 dB we can expect ±40, for SNR = 30 dB we can expect ±12.6, and for SNR = 40 dB we can expect ±4, where the full brightness range of the system is 256 units. Experiments were performed to show the effect of varying each of the detector parameters while keeping the other two constant. First, the value of r for the Gaussian smoothing filter was varied while the upper and lower thresholds of the detector were set at 0.7 and 0.3 respectively. Fig. 8 shows TP% and FP% against a range of r. A high r results in better noise reduction, but at the expense of more edge smoothing and a higher computational load. For the 30 and 40 dB SNRs, r P 1 results in a TP% greater than 83, whereas to reduce the FP% below 5, r P 1.5 is indicated. For a SNR of 10 dB, r > 2.5 was found to increase TP% to over 70 and to reduce FP% below 8. A TP% of 85 was the maximum found, because the non-maximal suppression stage, which limits the detected edge to be one pixel thick, does not always result in the retention of the best pixel to match the positions marked in the ground truth. Similarly the FP% never falls below 4%. Thus for high SNR images the FPs tended to lie close to
1616
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
90 40db
80
30db
70
20db
TP% and FP%
60 10db
50 40
10db
30 20db
20 10
30db 40db
0
0
0.5
1
1.5
2
2.5
3
Sigma
Fig. 8. TP% (—) and FP% (- - -) vs. r, for SNR from 10 dB to 40 dB.
the ground truth positions, whereas for low SNR images the FPs were randomly distributed throughout each image and resulted where noisy responses exceeded the upper-threshold. In the second simulation, the upper threshold was varied while the lower threshold and r were constant, and in the third simulation, the upper threshold was varied while the lower threshold and r were constant. Fig. 9 shows TP% against variation from 0.1 to 0.9 of both the upper threshold with the lower constant at 0.1, and the lower
threshold with the upper constant at 0.9. Smoothing was disabled (r = 0). The results show a small increase in TP% for a decreasing upper threshold, however a large increase can be obtained if the lower-threshold is reduced below 0.8 for high SNR images, below 0.6 for low SNR images, and below 0.3 for very low SNR images. Fig. 10 shows FP% against variation from 0.1 to 0.9 of both the upper-threshold with the lower constant at 0.1, and the lower-threshold with the upper constant at 0.9. Smoothing was disabled. For a high SNR the FP% can be kept below 5 if an upper-threshold greater than 0.9 is used. The lower-threshold is then immaterial. For the low SNR cases, the upper-threshold should again be kept above 0.9, and the lower-threshold above 0.8 to keep the FP% below 12. For this acquisition system with high SNR (30 dB) images and no smoothing filter, Figs. 9 and 10 indicate an upper-threshold of 0.9 and a lower of 0.8 will give good results, with TP% = 83.6 and FP% = 3.9 achieved. Fig. 8 indicates that smoothing (r = 1.0) should improve these results, and was found to increase TP% to 84.6 and reduce FP% to 3.7. However, a conflict occurs when Figs. 9 and 10 are used to estimate the lower-threshold for low SNR images, where a low lower-threshold results in a high TP%, but also a high FP%. For example with a
50
90 40db
45
80
10db 10db
30db
40 20db
70 35 20db
30
FP%
TP%
60 50
10db
40
25
20db 30db
20 15 40db
30 10
20
5
30db 40db
10 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Threshold
Fig. 9. TP% vs. upper (—) and lower threshold (- - -) vs. r, for SNR from 10 dB to 40 dB.
0 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Threshold
Fig. 10. FP% vs. upper (—) and lower threshold (- - -) vs. r, for SNR from 10 dB to 40 dB.
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
1617
1 shows optimum detector settings for the tested system.
Table 1 Optimum detector settings for tested system SNR (dB)
r
Lower threshold
Upper threshold
TP%
FP%
20 30
1.8 1.0
7.5 0.8
0.9 0.9
83.4 84.6
4.0 3.7
5.1. Example of use
SNR = 20 dB, Fig. 9 indicates a lower-threshold of 0.6 to maximize TP% at 60.4 (FP% = 14.3), but Fig. 10 indicates a threshold of 0.9 to minimize FP% at 6.9% (TP% = 44.5). The final choice will be problem dependant, but Fig. 8 indicates that improvement can be achieved in both TP% and FP% by increasing r to 1.8. Then by setting an intermediate lower-threshold of 7.5, TP% = 83.4 and FP% = 4.0 were obtained. Table
The acquisition system measured in the first part of Section 5 was used to image a soccer ball made from stitched panels. The image is displayed in Fig. 11(a). The characteristics displayed in Figs. 8–10 were used to choose parameters for the Canny Edge Detector. The characteristics were compiled from noise free data to which random white noise was added to give results for the various SNRs. For this test the standard deviation (SD) of the systemÕs intensity noise was measured so that the actual SNR of the image could be calculated. The SD varies with the intensity of
Fig. 11. Soccer ball edge maps. (a) Original, (b) r = 0.5, LT = 0.8, and UT = 0.9, (c) r = 1.0, LT = 0.8, and UT = 0.9, (d) r = 2.7, LT = 0.2, and UT = 0.7.
1618
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
the signal, and SD-vs.-intensity calibration curves can be compiled, Healey and Kondepudy (1994). However, here just the soccer ball image was considered, and the SD of the noise in the light and dark image areas measured using 64 timesequenced images of the scene to estimate the SD for each image pixel, Wittels et al. (1988). An SD = 1.8883 was the maximum found for the light areas, and SD = 1.4082 for the dark areas. Examining the image we want the detector to find edges around the ball and between the panels. The panels are stitched and then folded so we can expect up to three parallel edges at each join. The average intensity step height between the ball and the background was measured to be 120 units. Using Eq. (2), with SD = 1.8883 resulted in a SNR = 36 dB. Fig. 8 was used to estimate a suitable range of r parameter values. The 30 dB curve was used because it represents the next worst case below 36 dB. Considering true positive edge points, 0.5 6 r 6 1.0 will result in 80 6 TP% 6 85, and 4 6 FP% 6 12. This range of FP% is large, but r = 0.5 resulted in a more compact filter support and was chosen for the first test. Fig. 9 was used to estimate suitable lower (LT) and upper (UT) threshold values, with LT 6 0.8 indicating a TP% = 80. This constrained UT P LT. Fig. 10 was used to refine the thresholds and ensure a low FP%. It indicated UT = 0.9 to reduce FP% to 5 and that any LT can be used. Finally r = 0.5, LT = 0.8, and UT = 0.9 was chosen, and the resulting edge-map is shown in Fig. 11(b). There was no ground truth for this image, but from Figs. 8–10 TP% = 80 and FP% = 12 were estimated. The outline of the ball appears to be correct, but there are many noiseedge-points as might be expected with a high FP%. The experiment was repeated with r = 1.0. We expected FP% to reduce to 4, and the results in Fig. 11(c) now show an almost noise free background. Now, considering the missing edge between two of the panels in the left of the ball in Fig. 11(c), we can use Figs. 8–10 to estimate detector parameters to optimally detect this edge. The average step height was measured to be 7. The areas are both dark so the noise SD = 1.4082. This resulted in a SNR = 14 dB, a very low figure. The nearest
SNR to this, SNR = 10 dB, was used with Fig. 8 to identify that r = 2.7 will give TP% = 72 and FP% = 7. Fig. 9 indicates UT 6 0.7 and LT 6 0.2 to maximize TP%, whereas Fig. 10 indicates 0.7 6 UT 6 0.9 and LT as high as possible to minimize FP%. The final choice was r = 2.7, LT = 0.2, and UT = 0.7, and the resulting edge-map is shown in Fig. 11(d). The edges between the panels have now been partly detected. It would have been possible to determine the detection parameters by trial and error, but using the proposed method TP% and FP% estimates have been made available and can be maximized for a particular edge.
6. Conclusions Sets of realistic test edges with accurately known positions were produced and used to measure the performance of the Canny detector. The edges were realistic in that they were synthesized by convolving a step edge with the measured PSF of the particular image acquisition system under investigation. The test edge images were identical with those which would be captured by the acquisition system when viewing an object, but with the advantage that the actual positions of the edges were known and available for comparison with the detected result. With this technique it was also possible to allow a synthesized edge to locate to different sub-pixel positions and orientations within the central image pixel, and thus a more comprehensive set of tests were able to be performed. The detector results were expressed in a true positive and false positive format. The optimum values of the detectorÕs parameters were found for a series of edges to which increasing levels of noise had been added. Overall the Canny detector was found to work well. With low noise images, a few extra edge points were detected connected to the edges that were not part of the ground truth line. However with high noise images, extra edge points were detected randomly distributed within the image. The method will improve the setting up procedure of image acquisition systems. Given minimum required true positive and maximum false positive system per-
R.C. Staunton / Pattern Recognition Letters 26 (2005) 1609–1619
formance figures, the detector parameters can be chosen to minimize r for the Gaussian filtering stage, and thus maximize computational efficiency.
References Bowyer, K., Kranenburg, C., Dougherty, S., 2001. Edge detector evaluation using empirical ROC curves. Comput. Vision Image Underst. 84, 77–103. Canny, J., 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8 (6), 679–698. Davies, E.R., 1984. Circularity a new principle underlying the design of accurate edge orientation operators. Image Vision Comput. 2 (3), 134–142. Davis, L.S., 1975. A survey of edge detection techniques. Comput. Graphics Image Process. 4, 248–270. Healey, G.E., Kondepudy, R., 1994. Radiometric CCD camera calibration and noise estimation. IEEE Trans. Pattern Anal. Mach. Intell. 16 (3), 267–276.
1619
Lyvers, E.P., Mitchell, O.R., 1988. Precision edge contrast and orientation estimation. IEEE Trans. Pattern Anal. Mach. Intell. 10 (6), 927–937. Reichenbach, S.E., Park, S.K., Narayanswamy, R., 1991. Characterizing digital image acquisition devices. Opt. Eng. 30 (2), 170–177. Staunton, R.C., 1998. Edge operator error estimation incorporating measurements of CCD TV camera transfer function. IEE Proc. Vision Image Signal Process. 145 (3), 229–235. Tzannes, A.P., Mooney, J.M., 1995. Measurement of the modulation transfer function of infrared cameras. Opt. Eng. 34 (6), 1808–1817. White, R.G., Schowengerdt, R.A., 1994. Effect of point spread functions on precision step edge measurement. J. Opt. Soc. Am. Part A—Opt. Image Sci. Vis. 11 (10), 2593– 2603. Wittels, N., McClelland, J.R., Cushing, K., Howard, W., Palmer, A., 1988. How to select cameras for machine vision. Proc. SPIE 1197, 44–53.
Pattern Recognition Letters 26 (2005) 1620–1631 www.elsevier.com/locate/patrec
A fast and robust feature-based 3D algorithm using compressed image correlation Sheng S. Tan *, Douglas P. Hart Mechanical Engineering Department, Massachusetts Institute of Technology, 77 Mass Avenue, 3-243, Cambridge, MA 02139, USA Received 19 March 2004; received in revised form 19 December 2004 Available online 19 April 2005 Communicated by E. Backer
Abstract Two objectives of 3D computer vision are high processing speed and precise recovery of object boundaries. This paper addresses these issues by presenting an algorithm that combines feature-based 3D matching with Compressed Image Correlation. The algorithm uses an image compression scheme that retains pixel values in high intensity gradient areas while eliminating pixels with little correlation information in smooth surface regions. The remaining pixels are stored in sparse format along with their relative locations encoded into 32-bit words. The result is a highly reduced image data set containing distinct features at object boundaries. Consequently, far fewer memory calls and data entry comparisons are required to accurately determine edge movement. In addition, by utilizing an error correlation function, pixel comparisons are made through single integer calculations eliminating time consuming multiplication and floating point arithmetic. Thus, this algorithm typically results in much higher correlation speeds than spectral correlation and SSD algorithms. Unlike the traditional fixed window sorting scheme, adaptive correlation window positioning is implemented by dynamically placing object boundaries at the center of each correlation window. Processing speed is further improved by compressing and correlating the images in only the direction of disparity motion between frames. Test results on both simulated disparities and real motion image pair are presented. 2005 Elsevier B.V. All rights reserved. Keywords: Compressed image correlation; Gradient-based compression; Feature matching; Adaptive window; 3D vision
1. Introduction *
Corresponding author. Tel.: +1 617 253 0229; fax: +1 240 214 8665. E-mail address: [email protected] (S.S. Tan).
Features such as edges and corners play an important role in human vision. The visual cortex is especially responsive to strong features in a
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.009
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
scene (Sharma et al., 2003). Together with related abilities such as correspondence matching and tracking, humans are able to react quickly to the environment and focus attention on objects of interest. The significance of features is fully recognized in computer vision. For example, one traditional class of techniques applied to facial recognition is based on the computation of a set of geometrical features from a picture of the face such as the sizes and relative positions of eyes, mouth, nose and chin (Brunelli and Poggio, 1993; Gao and Leung, 2002). There is even belief that edge representations may contain all of the information required for the majority of higher level tasks (Elder, 1999). Feature-based 2D tracking is extensively implemented in automated surveillance, robotic manipulation and navigation. Because real-time processing is a necessity in these applications, only perceptually significant information such as contours is retained from video feeds. If the targetÕs 3D model is known, its detected contours are compared against its geometrical model to determine the objectÕs current position and orientation (Drummond and Cipolla, 2002). If there is no a priori knowledge of the target, it is tracked by finding the contoursÕ disparity between frames using cross correlation (Deriche and Faugeras, 1990) or level set (Mansouri, 2002). Passive 3D imaging can be reduced to the problem of resolving disparities between image frames from one or several cameras. Some key issues involved are lack of texture, discontinuity and speed. Numerous algorithms have been proposed to address these issues that fall into three broad categories: feature-based, area-based and volume-based algorithms (Scharstein and Szeliski, 2002). Same as in 2D tracking, feature-based 3D imaging techniques are able to process an extensive amount of video data in real-time while providing enough latency for high level tasks such as object recognition (Hoff and Ahuja, 1989; Olsen, 1990). This group of methods generates sparse but accurate depth maps at feature points and excels at determining object boundary position where area-based techniques often fail. When a full-field depth map is desirable, the sparse 3D representation provides a solid foundation for additional area- or volume-
1621
based algorithms to fill in the voids when there is ample surface texture; otherwise, when texture is scarce or highly repetitive, object segmentation methods and interpolation are preferable (Izquierdo, 1999; Mahamud et al., 2003). In the emerging field of image-based 3D editing, which has many applications in architectural design and entertainment, long and tedious human efforts are required to manually extract layers and assign depths for a 2D image (Oh et al., 2001). Automatic feature-based depth detection would greatly facilitate this process. Broad adoption of 3D imaging technology is currently limited by speed and robustness. Applications such as robotic surgery (Hoznek et al., 2002) and autonomous navigation or tracking, demand real-time processing. As an example, a texture mapped 3D scene would greatly aid in a surgeonÕs tactile sense. 3D reconstructed views enable better object recognition without turning the camera and taking more images. 3D object tracking would be much more robust than its 2D counterpart if reliable depth information is available. 1.1. Relation to prior work Current methods of finding feature correspondence can be categorized into global or local techniques. Global approaches to the sparse correspondence problem handle the entire set of sparse points by forming a global optimization function. Various constraints such as color constancy, continuity, uniqueness and epipolar constraints guide the search of a global solution. (Bartoli et al., 2003; Maciel and Costeira, 2003). Global techniques are usually robust but relatively slow due to the iteration and optimization process. Local methods find each pixelÕs correspondence by computing a cost function in a small interrogation window around the pixel of interest. Popular cost functions include SSD (Sum of Squared Differences) and cross correlation. Sparse Array Image Correlation is commonly implemented in the field of Particle Image Velocimetry (PIV), where fluid fields are seeded with fluorescent tracer particles and illuminated with a laser sheet. Flow motion is measured by tracking particle displacement. PIV images are comprised of millions of bright spots
1622
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
over a dark background. Each image is compressed into a subset of pixels before correlation that only include high gradient areas (Hart, 1998). This technique is especially fast and robust at handling large data sets. The algorithm presented in this paper shares the same computational grounds as Sparse Array Image Correlation. Depth discontinuities have been a major concern in area-based stereo matching. Boundary overreach, where the detected boundary locations deviate from the real boundaries, often occurs when the interrogation window contains both the boundary and its adjacent smooth surfaces. Adaptive window techniques have been developed to solve this problem (Kanade and Okutomi, 1994; Okutomi et al., 2002). An asymmetrical window is set around the pixel of interest so that the interrogation window does not cover the object boundary. A cost function is calculated for each possible window location around the pixel of interest and the window with optimized result is chosen. The disadvantage of such adaptive window schemes is that computational load is increased by an order of magnitude due to traversing in all the possible windows. 1.2. Contribution of this paper Speed and precise recovery of edge locations and disparities are the two main goals of the algorithm presented here. The purpose of edge detection in the algorithm is to aid in the compressed correlation process, which distinguishes itself from the typical correlation or SSD-based techniques. The remaining pixels are stored in sparse format along with their relative locations encoded into 32-bit words. Compression dramatically increases speed because only a fraction of the original pixels are retained for correlation. By introducing the well-established gradient-based compressed image correlation algorithm from the computational fluids community to the computer vision field, real-time scene reconstruction may gain new momentum. Coarse correlation is first performed to obtain an integer-pixel disparity estimation using large adaptive interrogation windows. Then fine correlation with smaller windows resolves each on-edge
pixelÕs disparity to sub-pixel resolution based on the rough estimation from coarse correlation. Error correlation is chosen over standard cross correlation because pixel comparisons are made through simple integer calculations rather than the computationally expensive multiplication and floating point arithmetic. In order to avoid the boundary overreach problem, adaptive window positioning is also utilized. However, in the proposed algorithm interrogation window selection is integrated with edge detection. The optimum window location is explicitly determined at the moment an edge is detected without the need of testing through a series of possible windows. In this paper, Section 2 describes the intensity gradient compression methodology and the significance of threshold setting. Section 3 explains error correlation in a compressed format. These two sections provide the computational grounds of the proposed algorithm. Section 4 shows how the appropriate window location is adaptively selected and its advantages. In Section 5, fine correlation combined with depth-based segmentation and interpolation is presented as a possible approach to generate a complete depth map. Section 6 provides experimental results of both simulated disparities and real image pairs. The quality of the disparity maps obtained demonstrates the effectiveness of the proposed algorithm.
2. Image compression The first step in compressed image correlation is to generate a data array that contains just enough information to determine the disparity between two images. From the statistical correlation function, PM PN m¼1 n¼1 ½I mþDi;nþDj I m;n ffi UDi;Dj ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi PM PN 2 ffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi PM PN 2 I I m¼1 n¼1 m;n m¼1 n¼1 mþDi;nþDj ð1Þ It is clear that pixels of low intensity contribute little to the correlation coefficient while pixels with high intensities have a much more significant weight due to squaring. This is the reason why
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
1623
cross correlation produces spurious vectors when there is a flare in one image caused by environmental lighting fluctuations. Correlation also fails in featureless, low intensity gradient regions where camera noise becomes significant. Much of the sub-pixel accuracy in image disparity comes from the pixels residing on edges. Thus, discarding low intensity areas and taking into account only strong features that are relatively insensitive to noise, improves correlation robustness. In the algorithm presented herein, local spatial gradients are calculated for each pixel by comparing the intensities of every other pixel instead of two neighboring pixels in order to preserve a wider edge. For gradients in both horizontal and vertical directions, the local gradient is approximated as: gradði; jÞ ¼ jIði þ 2; jÞ Iði; jÞj þ jIði; j þ 2Þ Iði; jÞj
ð2Þ
If disparities between two images only occur in a known direction, for example, horizontally, the gradient formula is reduced to: gradði; jÞ ¼ jði þ 2; jÞ Iði; jÞj
ð3Þ
For the sake of simplicity, this paper only deals with the horizontal disparity case. When the local gradient is larger than a preset threshold, e.g., 20 grayscales, the pixel of interest is retained and saved into a sparse array comprised of 32-bit long words along with its relative locations. Each 32-bit long word is divided into three sections: the last 8 bits store the pixel intensity, the middle 12 bits the y-index j and the first 12 bits the x-index i. For example, a pixel of intensity I = 60 at location i = 1078 and j = 395 is saved as 000110001011 010000110110 00111100 binary. Storing data in this compressed format significantly reduces the number of memory calls that must be made during correlation. The values of i, j and I can be quickly retrieved in a couple of CPU clock cycles by bit-shifting which is optimized for speed in most processors. The above gradient criterion is chosen for its simplicity and small region of support. The major concern here is extracting high intensity gradient pixels for correlation, not a complete edge map of enclosed contours. Other popular edge-detec-
Fig. 1. (a) The original right image. (b) Compressed image at threshold = 5 grayscales. Data retained = 24.1%. (c) Compressed image at threshold = 15 grayscales. Data retained = 2.86%.
tors such as Canny, Sobel and Gaussian are not only computational expensive but also require global filtering before edge detection (Horn, 1986). This is impossible for sparse array format where a simple block transfer cannot be done as in uncompressed format correlation. A proper threshold is essential to both speed and robustness. The higher the threshold the faster the algorithm because less pixels are stored in the sparse array for correlation; also better robustness because only major object boundaries are detected and minor image textures are omitted. The overall compression ratio is determined by both the image complexity and threshold. Fig. 1 illustrates the thresholdÕs role in extracting strong features. At a low threshold, not only the object boundaries but also texture-less areas such as the wall and table are detected. At a higher threshold only the clean edges are extracted.
3. Cross correlation in compressed format For each correlation window in the second image, the local gradient is first checked at each pixel in the window and qualified pixels are stored
1624
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
in compressed format. If the total number of pixels retained is small, it is considered an empty region and this window is discarded without correlating. If the number is large, e.g., 9 px in a 32 · 32 px window, then each pixel in the same correlation window in the first image is immediately compressed and cross correlated against the saved sparse array of the second image before the program moves on to the next window. This compressed correlation technique is most efficient when there is a minimum amount of overlap among interrogation windows. If there is significant overlap, the number of redundant memory calls and arithmetic calculation from repetitive gradient checking and correlation entries greatly slows processing. Error correlation is implemented rather than the traditional statistical correlation function because it replaces multiplication with the much faster addition and subtraction. In addition to being faster, it does not place an unduly significant weight on the high intensity pixels as does the statistical correlation function. It is shown that error correlation significantly improves processing speed while maintaining the level of accuracy compared to the statistical correlation function (Roth et al., 1995). The 2D error correlation function can be expressed as: UDi;Dj PM PN ½I m;n þ I mþDi;nþDj jI m;n I mþDi;nþDj j ¼ m¼1 n¼1 PM PN m¼1 n¼1 ½I m;n þ I mþDi;nþDj ð4Þ The 1D error correlation function in the horizontal direction is simplified to: PM ½I m þ I mþDi jI m I mþDi j UDi ¼ m¼1 PM ð5Þ m¼1 ½I m þ I mþDi While the typical statistical correlation function computes one entry at a time, error correlation is calculated at the same time as the sparse array is being generated. The entire correlation table is constructed by summing entries as they are found in one interrogation window while traversing through the sparse image array generated from the other corresponding interrogation window. The resulting disparity is obtained by searching
for the peak in the correlation coefficient plane. Simple bilinear interpolation is used to determine the correlation maximum within subpixel resolution. Compressed error correlation gives a very steep peak, which is ideal for bilinear interpolation.
4. Adaptive window positioning In the traditional fixed window position scheme, the entire image is evenly divided into uniformly spaced correlation windows, with or without overlapping. Such a window sorting method is easy to implement. However, it produces more spurious vectors than adaptive window approaches. Gross errors occur when an edge sits across two fixed correlation windows (Fig. 2). Fig. 3 is a real example. The edge in the first image is shifted by 8 px to the right. Both the original images and their corresponding extracted edge maps of two neighboring fixed correlation windows are shown. The two windows cut the edge in the second image. As a result, the measured disparity in the left window tends to be smaller than the true disparity because the edge is fully present in the first image and thus has a higher weight in the correlation table. Following the same logic, the right window gives a larger disparity estimate. Correlation result of the left two blocks is 7.23 px, and the right two blocks 8.88 px. In contrast, adaptive window positioning technique enhances robustness by intelligently placing the edges about the center of each correlation window. Each window is dynamically selected at the time an edge is detected. A searching scheme is devised so that when an edge pixel is extracted in the
Fig. 2. Demonstration of a scenario when an edge sits across two fixed neighboring correlation windows.
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
Fig. 3. An edge is shifted by 8 px to the right inside an image pair. The top shows the original images and the bottom their corresponding extracted edge maps. The two fixed neighboring windows cut the edge in the second image. Correlation of the left two blocks gives a disparity of 7.23 px, while the right gives 8.88 px.
second image, a correlation window is immediately placed around this pixel. All the pixels in this window are now accounted for and no more searching for additional interrogation windows is performed in this block in order to maximize speed and minimize window overlap. The edge in Fig. 3 is covered by only one correlation window using adaptive window positioning (Fig. 4) rather than two as with fixed windows (Fig. 3). Consequently the correlation result of this single window gives the correct 8 px.
Fig. 4. Adaptive window positioning is applied to the image pair in Fig. 3. The top shows the original images and the bottom their corresponding extracted edge maps. The one dynamically positioned correlation window holds the complete edge in both images. Correlation of this one block gives the true disparity of 8 px.
1625
The following example demonstrates the effectives of adaptive windows over fixed ones. The first image in the image pair is captured by a camera with an image size of 500 · 500 px (Fig. 5). The second image is obtained by simulating a uniform lateral disparity of 8 px relative to the first image. Integer disparity simulation is simply obtained by pixel index shifting. Compression and correlation are only performed in the X direction using the proposed edge matching algorithm as there is no vertical shift. The measured disparity results using fixed correlation windows are illustrated in Fig. 6. The interrogation window size is 32 · 32 px. Gradient threshold for compression is set at 20 grayscales. Notice that a number of edges are positioned across neighboring windows in the second image. The measured sparse disparity field has a mean of 7.67 px and a standard deviation of 1.56 px. The measured disparity results based on adaptively selected correlation windows are illustrated in Fig. 7. Each cross correlation window location is determined based on the edge map of the second image. Note the significantly improved accuracy with adaptive windows. The measured disparity field of valid vectors has a mean of 7.99 px and a standard deviation of 0.0505 px. The time required for both block finding and compressed correlation is 7.5 ms on a Xeon 2.8 GHz desktop using C++. For the purpose of statistical analysis, vector validation threshold is set at ±0.5 px from the true disparity. Any measured disparity that falls
Fig. 5. Left: the original image of several building blocks captured by a camera. The image size is 500 · 500 px. Downloaded from the CMU Image Database at http://vasc.ri.cmu.edu//idb/html/stereo/arch/. Right: the second image is artificially shifted to the right by 8 px.
1626
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
Fig. 6. Compressed coarse correlation results of an image pair with a simulated horizontal disparity of 8 px using fixed windows. The edge map of the second image is shown. Each block represents a non-empty correlation window. Interrogation window size is 32 · 32 px. Gradient threshold for compression is 20 grayscales. The measured disparity field has a mean of 7.67 px with a standard deviation of 1.56 px.
Fig. 8. Demonstration of the image boundary effect. Gross error occurs when some edges are entering or leaving the field of view between exposures. The top shows the original images and the bottom their corresponding extracted edge maps. The measured disparity is 5.91 px compared to the ground truth of 8 px.
outside this tolerance range is classified as an outlier; otherwise, a valid vector. For example, if the true disparity = 8 px, the range of valid measured vectors is 7.5–8.5 px. If the true disparity = 1 px, valid range of measured vectors is 0.5–1.5 px. The only invalid vector in Fig. 7 is due to the image boundary effect. Gross error occurs when some edges are entering or leaving the field of view between exposures. This issue is probably unsolvable with only two images. Fig. 8 shows both the original images and extracted edge maps of the invalid correlation window in Fig. 7. The measured disparity is 5.91 px compared to the true disparity of 8 px.
5. Fine correlation and complete depth map
Fig. 7. Compressed coarse correlation results of an image pair with a simulated horizontal disparity of 8 px using adaptive windows. Correlation window location is determined based on the edge map of the second image. Interrogation window size is 32 · 32 px. Gradient threshold for compression is 20 grayscales. The measured disparity field of valid vectors has a mean of 7.99 px with a standard deviation of 0.0505 px. Number of valid vectors = 50. Number of invalid vectors = 1.
Coarse compressed correlation provides an averaged disparity estimate for each window. A large window size is necessary in order to accommodate edges of different shapes and orientations in both images. As explained in Section 4, cross correlation generates spurious vectors if an edge is fully present in one correlation window but half missing in the same correlation window in the other image. Usually, the coarse correlation window size is chosen to be roughly twice the size of the largest expected output disparity. In general,
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
a larger window produces fewer errors but it slows processing. Once a disparity estimate from coarse correlation is known, fine correlation can be performed for each on-edge pixel in the primary correlation window using a much smaller window size. The location of fine correlation window in the second image is set around the pixel of interest. The corresponding fine correlation window in the first image is shifted by the integer amount of the coarse correlation output, as illustrated in Fig. 9. Here, fine correlation window size is chosen to be 7 · 7 px. As a general rule, fine correlation speed dramatically improves with reduced fine correlation window size, at the cost of reduced accuracy. Other aspects of the fine compressed correlation are the same as coarse correlation except a lower gradient threshold. The compression threshold is lowered to avoid the loss of any useful information in the reduced correlation window. This practice does not compromise robustness since a valid disparity has been identified using a higher threshold in coarse correlation. Fig. 10 illustrates the fine correlation results up to single pixel resolution based on the coarse correlation output as shown in Fig. 7. Fine correlation has an improved accuracy over coarse correlation because of the window shifting. In this simulated disparity case, the disparity of every on-edge pixel is correctly recovered with a standard deviation of 0 px. In some applications such as 3D feature-based facial recognition (Gao and Leung, 2002) and ob-
1627
Fine correlation results 9
8.5
8
7.5
7 0 100 200 300 Y (pixel)
400
0
300
200
100
400
500
X (pixel)
Fig. 10. Sparse fine correlation results up to single pixel resolution1 of an image pair with a simulated disparity of 8 px. Fine correlation window size is set at 7 · 7 px, and threshold = 15 grayscales. The sparse disparity field has a mean of 8 px with a standard deviation of 0 px.
L
Search for the next coarse correlation window in Image 2
R
Original image pair N
# of pixel retained > min required? Y Compressed coarse correlation Compressed fine correlation for each on-edge pixel
Entire image searched?
N
Y Y Fig. 9. Example of a fine correlation window selection in a coarse correlation window with a calculated disparity of 8 px. The pixel of interest is at the center of the fine correlation window in the second image. The corresponding fine correlation window in the first image is shifted to the left by 8 px. Fine correlation window size is 7 · 7 px.
Correlate empty regions at a reduced threshold
Sufficient texture?
N
Depth-driven segmentation and interpolation
Fig. 11. Overall flow of the proposed algorithm.
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
ject tracking (Drummond and Cipolla, 2002), a sparse disparity map of object boundaries provides enough information. In other applications such as 3D scene reconstruction, a complete disparity map is preferable. There are two general approaches to fill the regions of unknown disparities among the recovered edges. If there is sufficient surface texture, compressed cross correlation at a lower threshold can be performed in such areas. However, in many real world cases such as the scene in Fig. 1, there is a lack of fine texture, which is critical to a reliable correlation. Instead, depth-driven object segmentation and interpolation is necessary to obtain a full field disparity rendering. The overall flow of the proposed edge-matching algorithm with complete disparity map output is shown schematically in Fig. 11.
Standard deviation of measured disparities 0.16
0.14
Standard deviation (± pixel)
1628
0.12
0.1
0.08
0.06
0.04
0
2
4
6
8
10
Measured disparity (pixel)
Fig. 12. Standard deviation of measured disparities of a sequence of images with a simulated horizontal disparity from 0.2 to 11 px relative to the original image. Coarse correlation window size = 32 · 32 px. Fine correlation window size = 7 · 7 px and threshold = 15 grayscales.
6. Experiments 6.1. Simulated disparity The original image in Fig. 5 is artificially shifted laterally to the right from 0.2 to 11 px at an interval of 0.2 px. Sub-pixel disparity is approximated using the shift theorem in the frequency domain. Fine correlation is calculated at each on-edge pixel. Fig. 12 shows the standard deviation of fine correlation results throughout the entire sequence of lateral disparities. The maximum standard deviation among integer disparities is ±0.0812 px. The maximum error among simulated sub-pixel disparities is ±0.147 px. Overall mean value of standard deviations is ±0.098 px. Given the pixel intensity rounding error introduced in the sub-pixel shifting simulation, it is fair to conclude that the proposed algorithm has an accuracy upper limit of ±0.1 px, which is consistent with the best obtainable accuracy from interpolating a single correlation plane calculated with only two images. 6.2. Real motion image pair In the following ‘‘MIT’’ scene as shown in Fig. 16, both images in the image pair are captured by
a camera. The image size is 1152 · 864 px. The camera has a lateral displacement between the two exposures resulting in right-hand disparities in the image plane. The three objects are placed on various depth planes. The closer the object is to the camera, the larger its disparity between frames due to different magnification ratios. Choosing a proper coarse correlation window size is critical to the proposed algorithmÕs performance on speed and accuracy. Overall, a larger window size produces fewer gross errors but it slows processing because the correlation load increase as the square of the window size. On the other hand, smaller correlation window size results in higher spatial resolution and thus less averaging effect in areas where a number of objects at different depths are close to each other. In the following example, a rectangular window shape is able to take advantage of both large and small window sizes since disparities are known to occur in only one direction. The horizontal correlation window size is chosen to be 64 px, while the vertical window size is 8 px. Fig. 13 shows the vector field of coarse correlation results as well as their adaptively selected corresponding window. Fig. 14 provides a side view of the coarse correlation disparity
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
1629
Fig. 13. Coarse correlation results of a motion image pair with lateral disparity. CorrSizeX = 64 CorrSizeY = 8, threshold = 15.
16 14
Disparity (pixel)
12 10 8 6
Fig. 15. Fine correlation results calculated based on the coarse correlation output shown in Fig. 13. (a) top view; (b) front view.
4 2 0
100
200
300
400
500
600
700
800
900
Y (pixel)
Fig. 14. Side view of the coarse correlation disparity field shown in Fig. 13.
field. The processing time of both blocks finding and coarse correlation is 29.0 ms on a Xeon 2.8 GHz desktop for an image pair with a size of 1152 · 864 px. Fig. 15 illustrates both the top and front views of the fine compressed correlation disparity field comprised of all the detected onedge pixels. After a fine disparity map of the object boundaries is acquired, a simple segmentation and interpolation algorithm is implemented to fill the voids
among the edges since there is no sufficient texture on smooth surfaces for a reliable correlation. Fig. 16 shows the full disparity field rendering of the ‘‘MIT’’ scene both with and without texture mapping. The results are encouraging considering only a single image pair is used as input. If there is sufficient texture on smooth surfaces, compressed correlation over the entire image plane can be performed to obtain a complete disparity map. Two different levels of compression will be implemented. In boundary regions, a strong compression is applied which results in the precise recovery of edges. In regions of small gradient variations, which often correspond to smooth surfaces, a mild compression is held, in which case
1630
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631
7. Conclusions In this paper, a new 3D algorithm has been proposed which can recover precise object boundaries at high speed by utilizing compressed image correlation and adaptive windows. Although the algorithm is relatively simple, the experimental results obtained are encouraging. An important feature to note is that this algorithm does not include any global optimization. Compressed image correlation is a technique by which stereo or motion image pairs can be accurately processed at high speeds. It is based on the compression of images in which the number of data set entries is reduced to containing only strong features. Very high correlation speeds are obtained by encrypting the reduced data set into sparse arrays and correlating the data entries using an error correlation function to eliminate time consuming multiplication, division and floating point arithmetic. The performance of this compressed image correlation, however, is largely dependant on image complexity. For applications requiring extremely high speeds such as real-time tracking and video rate stereo vision, the proposed edge-based 3D algorithm appears to be a viable processing technique. Future work includes integrating adaptive window shape and size into the proposed algorithm, as well as multiple image pairs. An extensive analysis of error rate vs. spatial frequency would fully demonstrate its effectiveness and limitations.
References
Fig. 16. Full disparity field rendering of the ‘‘MIT’’ scene. (a) left image; (b) right image; (c) and (d) top view and 3D rendering of the complete disparity field after segmenting and interpolating the sparse correlation output shown in Fig. 15; (e) and (f) two views of the complete disparity field with texture mapping.
any useful information for correlation is retained including the minor features in surface texture.
Bartoli, A., Hartley, R., Kahl, F., 2003. Motion from 3D line correspondences: Linear and non-linear solutions. IEEE Internat. Conf. on Computer Vision and Pattern Recognition (CVPRÕ03), Madison, Wisconsin. Brunelli, R., Poggio, T., 1993. Face recognition: Features versus templates. IEEE Trans. Pattern Anal. Machine Intell. 15 (10), 1042–1052. Deriche, R., Faugeras, O.D., 1990. Tracking line segments. European Conference on Computer Vision (ECCVÕ90), Antibes, France, Springer 1990. Drummond, T., Cipolla, R., 2002. Real-time visual tracking of complex structures. IEEE Trans. Pattern Anal. Machine Intell. 24 (7), 932–946.
S.S. Tan, D.P. Hart / Pattern Recognition Letters 26 (2005) 1620–1631 Elder, J.H., 1999. Are edges incomplete? Internat. J. Comput. Vision 34 (2–3), 97–122. Gao, Y., Leung, M.K.H., 2002. Face recognition using line edge map. IEEE Trans. Pattern Anal. Machine Intell. 24 (6), 764–779. Hart, D.P., 1998. High-speed PIV analysis using compressed image correlation. J. Fluids Eng—Trans. ASME 120 (3), 463–470. Hoff, W., Ahuja, N., 1989. Surfaces from stereo: Integrating feature matching, disparity estimation, and contour detection. IEEE Trans. Pattern Anal. Machine Intell. 11 (2), 121– 136. Horn, B.K.P., 1986. Robot Vision. MIT Press, Cambridge, MA. Hoznek, A., Zaki, S., Samadi, D., Salomon, L., Lobontiu, A., Lang, P., Abbou, C.-C., 2002. Robotic assisted kidney transplantation: An initial experience. J. Urology 167 (4), 1604–1606. Izquierdo, M.E., 1999. Disparity segmentation analysis: Matching with an adaptive window and depth-driven segmentation. IEEE Trans. Circuits Syst Video Technol 9 (4), 589– 607. Kanade, T., Okutomi, M., 1994. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Trans. Pattern Anal. Machine Intell. 16 (9), 920–932. Maciel, J., Costeira, J.P., 2003. A global solution to sparse correspondence problems. IEEE Trans. Pattern Anal. Machine Intell. 25 (2), 187–199.
1631
Mahamud, S., Williams, L.R., Thornber, K.K., Xu, K., 2003. Segmentation of multiple salient closed contours from real images. IEEE Trans. Pattern Anal. Machine Intell. 25 (4), 433–444. Mansouri, A.-R., 2002. Region tracking via level set PDEs without motion computation. IEEE Trans. Pattern Anal. Machine Intell. 24 (7), 947–961. Oh, B.M., Chen, M., Dorsey, J., Durand, F., 2001. Imagebased modeling and photo editing. SIGGRAPH 2001. Okutomi, M., Katayama, Y., Oka, S., 2002. A simple stereo algorithm to recover precise object boundaries and smooth surfaces. Internat. J. Comput. Vision 47 (1–3), 261–273. Olsen, S.I., 1990. Stereo correspondence by surface reconstruction. IEEE Trans. Pattern Anal. Machine Intell. 12 (3), 309– 315. Roth, G., Hart, D.P., Katz, J., 1995. Feasibility of using the L64720 video motion estimation processor (MEP) to increase efficiency of velocity map generation for particle image velocimetry (PIV). ASME/JSME Fluids Engineering and laser Anemometry Conference, Hilton Head, South Carolina. Scharstein, D., Szeliski, R., 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Internat. J. Comput. Vision 47 (1–3), 7–42. Sharma, J., Dragoi, V., Tenenbaum, J.B., Miller, E.K., Sur, M., 2003. V1 Neurons signal acquisition of an internal representation of stimulus location. Sci. Mag. 300, 1758–1763.
Pattern Recognition Letters 26 (2005) 1641–1649 www.elsevier.com/locate/patrec
Robust face detection using Gabor filter features Lin-Lin Huang *, Akinobu Shimizu, Hidefumi Kobatake Graduate School of BASE, Tokyo University of Agriculture and Technology, 2-24-16 Naka-cho, Koganei-shi, Tokyo 184-8588, Japan Received 4 March 2004; received in revised form 30 November 2004 Available online 14 April 2005 Communicated by E. Backer
Abstract In this paper, we present a classification-based face detection method using Gabor filter features. Taking advantage of the desirable characteristics of spatial locality and orientation selectivity of Gabor filters, we design four filters corresponding to four orientations for extracting facial features from local images in sliding windows. The feature vector based on Gabor filters is used as the input of the face/non-face classifier, which is a polynomial neural network (PNN) on a reduced feature subspace learned by principal component analysis (PCA). The effectiveness of the proposed method is demonstrated by experiments on a large number of images. We show that using both of the magnitude and phase of Gabor filter response as features, the detection performance is better than that using magnitude only, and using the real part only also performs fairly well. Our detection performance is competitive with those reported in the literature. 2005 Elsevier B.V. All rights reserved. Keywords: Face detection; Classification; Gabor filter; Polynomial neural network
1. Introduction Machine recognition of human faces is motivated by wide applications ranging from static matching of controlled format photographs such as passports, credit cards and mug shots to realtime matching of surveillance video images (Chellappa et al., 1995). An automated face recognition
*
Corresponding author. Tel./fax: +81 42 388 7438. E-mail address: [email protected] (L.-L. Huang).
system often consists of three modules: face detection, facial feature detection and face recognition. Nevertheless, most existing methods assume that human faces have been located and focus on the recognition algorithm only. To build fully automated face recognition systems, it is essential to develop robust and efficient algorithms to detect faces regardless of pose, scale, orientation, expression, occlusion and lighting condition (Hjelmas and Low, 2001; Yang et al., 2002). The methods proposed for face detection so far generally fall into two major categories:
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.015
1642
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
feature-based methods and classification-based ones. Feature-based methods detect faces by searching for facial features (eyes, nose and mouth) and grouping them into faces according to their geometrical relationships (Lin and Fan, 2001; Wong et al., 2001). Since the performance of feature-based methods primarily depends on the reliable location of facial features, it is susceptible to partial occlusion, excessive deformation, and low quality of images. By classification-based methods, face detection is performed by shifting a search window over the input image and each local image in the window is classified to face/non-face class using a classifier. As such, the feature extraction from local images and the design of classifier are important to the detection performance. Many classification models have been proposed for face detection, such as Bayesian classifier (Yang et al., 2001; Liu, 2003), neural networks (Rowley et al., 1998; Sung and Poggio, 1998; Feraud et al., 2001), support vector machines (SVM) (Osuna et al., 1997; Heisele et al., 2003), etc. As to feature extraction, most classification-based methods have used the intensity values of the windowed image as the input features of the classifier. Since a pixel value does not account for any shape or texture characteristics, more effective features that incorporate neighborhood relationships should be exploited to further improve the detection performance. This kind of features can be extracted by spatial filtering using Gabor filters that are selective to both frequency and orientation. Gabor filters have been considered as a very useful tool in computer vision and image analysis due to its optimal localization properties in both spatial domain and spatial frequency domain. The impulse response functions of Gabor filters were shown to well model the receptive field profiles of visual cortical cells of mammalians and exhibit desirable properties of spatial locality and orientation selectivity (Daugman, 1988). The authors of Adini et al. (1997) indicated that face representation based on 2D Gabor filters is more robust against illumination variations in face recognition than intensity values. Gabor filter-based features have been applied to face recognition (Zhang et al., 1997; Liu and Wechester, 2002)
and facial expression classification (Donato et al., 1999) with great success. However, little work has been done to apply it to face detection. This paper proposes a classification-based approach using Gabor filter features for detecting faces from cluttered images. We design four Gabor filters corresponding to four orientations for extracting facial features from the local image in sliding window. The feature vector based on Gabor filters is used as the input of the classifier, which is a polynomial neural network (PNN) on a reduced feature subspace learned by principal component analysis (PCA) (Huang et al., 2003). The effectiveness of the proposed method is demonstrated by experimental results on testing a large number of images. In order to investigate the effectiveness of different part of Gabor filter response, three features are constructed based on the real part of Gabor filter response, magnitude, and the combination of magnitude and phase. The experimental results show the combination of magnitude and phase performs the best. The rest of this paper is organized as follows. Section 2 gives an overview of our face detection system; Section 3 describes the Gabor feature extraction method; Section 4 explains the PNN structure and the learning algorithm; The experimental results are presented in Section 5, and Section 6 provides concluding remarks.
2. System overview The system diagram is shown in Fig. 1. To detect faces of variable sizes and locations, the detector needs to examine shifted regions (scanned by a window) of the test image in multiple scales. The classifier is used to classify the local image in the window to one of two classes: face or non-face. To disambiguate overlapping detected regions, each local image is assigned a face likelihood measure, which should be high for a face region and low for a non-face region. The overlapping detected regions (candidate regions) within one scale or across different scales compete with each other such that only the candidate region with the highest face likelihood is retained as the detected face. In our work, the size of sliding window is 20 · 20 pixels.
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
1643
Fig. 1. Diagram of the face detection sytem.
The input image is re-scaled to multiple scales such that a face region is re-scaled to approximately 20 · 20 pixels in one of the re-scaled images. In order to alleviate the variation of lighting conditions, a pre-processing procedure similar to Sung and Poggio (1998) (subtracting each local image with an optimally fitted linear plane) is employed. Then the intensity values of the local image can serve as feature values for classification as in our previous work (Huang et al., 2003). To achieve better performance, more discriminative features can be extracted by spatial filtering using Gabor filters. We extract facial features using four Gabor filters corresponding to four orientations and construct feature vector as the input of the classifier. The classifier used in our system is a polynomial neural network (PNN), which uses the polynomial terms of features as the inputs of a single-layer network with one output unit for assigning face likelihood. To overcome the polynomial complexity of the PNN, which is formidable on large number of features, the dimensionality of the feature vector is reduced by PCA.
due to its optimal localization properties in both spatial and frequency domain. The general functional form of a 2D Gabor filter specified in space and spatial frequency domain is given by " !# 1 x02 þ y 02 gðx; yÞ ¼ exp 2pr2xy 2r2xy r2 expð2pir0 x0 Þ exp 02 ; 2ruv where 0 x ¼ x cos h þ y sin h; y 0 ¼ x sin h þ y cos h;
3.1. Gabor filter design Gabor filters have been applied to various image recognition problems for feature extraction
ð2Þ
where rxy is the standard deviations of the Gaussian envelope which characterizes the spatial extent and the bandwidth of the filter. The parameters (u0, v0) define the spatial frequency of a sinusoidal plane wave, which can also be expressed in polar coordinates as radial frequency r0 and orientation h: r20 ¼ u20 þ v20 ;
3. Gabor filter-based feature extraction
ð1Þ
tan h ¼
v0 . u0
ð3Þ
The frequency and orientation-selective properties of a Gabor filter are more explicit in its frequency domain representation Eq. (4) that specifies the amount by which the filter modifies or modulates each frequency component of the input image.
1644
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
( "
ðu u0 Þ2 þ ðv v0 Þ2 Gðu; vÞ ¼ exp 2r2uv 2 r exp 02 ; 2ruv 1 . ruv ¼ 2prxy
#)
Consequently, the radial frequency of Gabor filters is given by r0 ¼
3.2. Gabor feature extraction
pðk 1Þ ; k ¼ 1; 2; 3; 4. ð5Þ 4 The frequency responses of the four filters are depicted in Fig. 2. The frequency band of each filter is a Gaussian centered at (u0, v0). We determine the parameters of Gabor filters according to the sampling theorem (Castlman, 1996). When the sampling rate is 0.5 (we sample 10 · 10 values on filtered local images of 20 · 20 pixels), the maximum frequency rmax of Gabor filters is 0.25. In our work, the half-peak magnitude bandwidth is adopted, which is approximately 1.177ruv. Then we have:
hk ¼
where a ¼ 1.177;
rmax ¼ 0.25. ð6Þ
In addition, we can see from Fig. 2 that radial frequency r0 and ruv has the relation of: aruv ¼ r0 tanðp=8Þ.
ð8Þ
ð4Þ
The property of Gabor filter is defined by radial frequency r0, orientation and filter bandwidth. In this paper, Gabor filters are designed in the sense of the effectiveness of facial feature extraction. Specifically, in frontal or near frontal face image, despite the variation of face identity and expression, the facial contour is approximately an oval and the eyes and mouth are approximately in horizontal while the nose is in vertical. Therefore, Gabor filters with four orientations are considered:
r0 þ aruv ¼ rmax ;
1 4½1 þ 1.177 tanðp=8Þ
ð7Þ
Fig. 2. The four filters in the spatial frequency domain.
Gabor features of a local image in sliding window are extracted based on its Gabor representations, which are obtained by convolving the image with Gabor filters. Let I(x, y) be an image, the convolution of image I(x, y) and a Gabor filter is denoted as Oðx; y; r0 ; hk Þ ¼ Iðx; yÞ gðx; y; r0 ; hk Þ; k ¼ 1; 2; 3; 4;
ð9Þ
where O(x, y, r0, hk) is called as the Gabor representation of image I(x, y). Fig. 3 shows the Gabor representations of a face sample image (convolved with the real part of Gabor filter response). The images are visualized by coding the output values of Gabor filter in gray levels. We can see that the orientation properties of face pattern are well represented by the filtered images. In order to investigate the properties of different part of Gabor representation, three kinds of feature vectors, namely, consine, magnitude and mag + phase, are formed. Specifically, feature vector consine is constructed using the real part of Gabor representation Real{O(x, y, r0, hk)}. magnitude is computed as square root of the real part and the imagine part while feature vector mag + phase is formed by adding the phase of O(x, y, r0, hk) to feature vector magnitude. To encompass the orientation selectivities, we concatenate all the four orientation Gabor representations to construct feature vector. Before the concatenation, we down-sample the Gabor representation O(x, y, r0, hk) by a factor q = 2. In our work, the size of search window is 20 · 20 pixels so that each Gabor representation used for constructing feature vector has 10 · 10 pixel values after down-sampled. Therefore the dimensionality of both consine and magnitude is 400 while feature vector mag + phase has 800 dimensionalities.
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
1645
Fig. 3. A face sample image and its four Gabor filtered images.
4. Classification method For classification, we use a polynomial neural network (PNN) to assign a face likelihood to the sliding window so as to classify the window image into face or non-face. The feature vectors described in Section 3 are employed as the input of PNN. The PNN is a single-layer network which uses as inputs not only the feature measurements of the input pattern but also the polynomial terms of the measurements (Schu¨rmann, 1996). Compared with the multi-layer perceptron (MLP), the PNN is faster in learning and less susceptible to local minimum due to its single-layer structure. In our previous experiments, the PNN was shown to outperform the MLP in face detection (Huang et al., 2003). Denote the input pattern as a feature vector x = (x1, x2, . . ., xd)T, the output of the PNN is computed by ! d d X d X X yðxÞ ¼ s wi xi þ wij xi xj þ w0 ; ð10Þ i¼1
i¼1
j¼i
1 . 1 þ expðaÞ
In our problem, the dimensionality of input vector is high. To reduce the complexity of PNN for high-dimensional data, the dimensionality of the raw feature vector is reduced by PCA. The raw vector is projected onto a linear subspace: zj ¼ ðx lÞT /j ;
j ¼ 1; 2; . . . ; m;
i¼1
i¼1
j¼i
ð12Þ 2
Df ¼ kx lk
m X
z2j .
ð13Þ
j¼1
The connecting weights are initialized randomly and updated by stochastic gradient descent to minimize the empirical loss of mean square error (MSE) (Robbins and Monro, 1951): Nx Nx X X 2 2 E¼ ½yðxn Þ tn þ kkwk ¼ En ; ð14Þ n¼1
where s( Æ ) is a sigmoid activation function. sðaÞ ¼
tern reconstruction from the subspace is minimized. At mean time, the reconstruction error of feature space (the distance from the feature subspace, DFFS) is an important indicator of the deviation of the input pattern from the subspace. When the subspace is learned from face samples, the DFFS indicates the dissimilarity of a pattern being a face. Hence, we integrate the DFFS into the PNN: ! m m X m X X w i zi þ wij zi zj þ wD Df þ w0 ; yðxÞ ¼ s
n¼1
where tn denotes the target output for the input pattern xn, with value 1 for face pattern and 0 for non-face pattern; k is a coefficient of weight decay, which is helpful to improve the generalization performance. The PNN is trained on face and non-face samples. Since the PNN is a single-layer network, the training process is quite fast and not influenced by the random initialization of weights.
ð11Þ
where zj denotes the projection of x onto the jth axis of the subspace, /j denotes the eigenvector of the axis, and l denotes the mean vector of the pattern space. The eigenvectors corresponding to the m largest eigenvalues are selected such that the error of pat-
5. Experiments 5.1. Training sample collection The algorithm of training sample collection is the same as that in our previous work (Huang
1646
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
et al., 2003). We use 2987 images to extract face samples, which contain 2990 real faces. Each hand cropped face box is normalized into 20 · 20 pixels and gives one face sample. The local image within the box is stretched and compressed to give four samples. In addition, the mirror images of the above five face samples are also included. This is helpful to ensure that the final face samples cover the variations of face pattern in real world as much as possible. In total, 29,900 face samples are collected. Besides for training purpose, they were firstly used to compute the feature subspace by PCA. The non-face samples were collected with a preliminary classifier to identify the background patterns that resemble faces. The non-face samples were collected in three phases. In the first phase, the local images of search windows in background area are compared with the mean vector of face samples and the window with the Euclidean distance under a threshold is considered as a confusing non-face sample. 38,834 non-face samples are gathered. These samples and the 29,900 face samples are used to train the first-phase PNN. In the second and the third phase, the local window images are classified by the trained PNN and the local image which does not contain a face but have a output value higher than a threshold is considered as a confusing non-face sample. About 60,000 non-face samples are collected in the second and the third phrases. Finally, the PNN is re-trained with the face samples and all the nonface samples collected in three phases. 5.2. Experimental results We test the proposed method on two sets of images, which are totally different with those used in training procedure. Test Set1 consists of 109 CMU images,1 most of the images contain more than one face with cluttered background such that Test Set1 has a total of 487 faces. Test Set2 consists of 270 images with simple background and 1 The test set1 of Rowley et al. (1998) contains 130 images. We did not use the 7 images that contain extremely large or small faces and 14 scene images that have been used in training procedure.
each of them contains only one face, so totally there are 270 faces in Test Set2. The images of Test Set2 are downloaded from several websites. In our experiments, a face is successfully detected if the local image of search window contains both eyes and the mouth. Otherwise, it is a false positive. The detection rate is the ratio between the number of successful detection and the total number of faces in the Test set. The false positive rate is the ratio between the number of false positives and the total number of searching windows.
Fig. 4. ROC curves on Test Set1.
Table 1 Detection results on Test Set1 (109 images) Dimensionality
consine magnitude mag + phase
Detection rate (%)
False positives
False positive rate
86.4 86.4 86.4
88 79 57
1.1 · 106 1.0 · 106 7.6 · 107
Table 2 Detection results on Test Set2 (270 images) Dimensionality
consine magnitude mag + phase
Detection rate (%)
False positives
False positive rate
100 100 100
11 7 3
4.6 · 107 2.9 · 107 1.2 · 107
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
In detection, the images are resized in 10 scales. For images in Set1, the scale starts from 0.2 and increase at factor of 1.21. The images in Set2 have larger faces, so the scale is from 0.1. In our previous work (Huang et al., 2003), using image intensity as the raw feature (368D vector composed of intensity values of local image of
1647
search window undergone pre-processing procedure, referred to as Inten below), the PNN has yielded promising detection performance. To further improve the detection performance, we exploit Gabor filter features in this paper. Using the four kinds of feature vectors (Inten, consine, magnitude and mag + phase) as the input
Fig. 5. Detection examples on the images of Test Set1.
1648
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
of the PNN, we tested the 109 images of Test Set1 and drew the ROC (Receiver Operating Characteristics) curves of tradeoff between the detection rate and the false positive rate with variable decision thresholds. In order to make fair comparison, the dimensionalities of the subspaces employed by the four feature vectors are set to be the same, m = 100. The ROC curves is shown in Fig. 4. From the ROC curves, we can see that the Gabor filter features result in significant improvements compared to intensity features, which justifies that Gabor filter-based features are more effective than intensity-based features. This could be explained as that the orientation properties of face pattern captured by Gabor filters are visually prominent feature and are highly stable, therefore the Gabor filter-based features are more informative and robust to illumination and facial expression changes. Among the performances of three Gabor filter feature vectors, it is shown that the performance of mag + phase is the best while magnitude is superior to consine. The reason could be that the real part and imagine part of Gabor representation provide complementary information for classification. The detection rates and false positive rates on Test Set1 and Test Set2 are listed in Tables 1 and 2, respectively. We can see that on simple images of Test Set2, the detection rate is 100% and the false positive rate is very low. On Test Set1, the detection rate is lower. The reason is that the
images in Test Set1 mostly have lower resolution and more complex backgrounds than those in Test Set2. Some examples of face detection using consine are shown in Figs. 5 and 6. From these examples, we can see that the feature vector based on Gabor filters is quite robust against low image quality and face shape variation. The missed faces are inherently blurry or rotated excessively while the false positives mostly resemble face pattern when viewed in isolation. 5.3. Comparison of performance For comparison with other system, we would like to mention the method proposed by Sung and Poggio (1998), which models the distributions of face and non-face patterns and uses a multi-layer perceptron (MLP) to make final decision. The detection result of Sung and Poggio (1998) has been considered as one of the best results in detecting faces from images with complex backgrounds (Hjelmas and Low, 2001). The detection result of Table 3 Detection results on images of Sung and Poggio (1998) (23 images) Method
True positives
False positives
consine magnitude mag + phase Sung and Poggio (1998)
126 128 130 126
10 8 7 13
Fig. 6. Detection examples on the images of Test Set2.
L.-L. Huang et al. / Pattern Recognition Letters 26 (2005) 1641–1649
Sung and Poggio (1998) on a set of 23 cluttered images along with the result of our method on testing the same images are listed in Table 3. Although it is difficult to give a fair comparison between different methods due to the different training samples, training procedure and execution time, etc., based on the reported results, the detection performance of our method is comparable with that of Sung and Poggio (1998), yet with fewer false positives. In addition, for processing one search window (19 · 19 pixels), the classifier of Sung and Poggio (1998) needs more than 254,700 multiplications, while our method needs about 125,000 for one window of 20 · 20 pixels when using mag + phase. Therefore, our method consumes much less computation resources than that of Sung and Poggio (1998). 6. Conclusion This paper proposes a classification-based face detection approach using Gabor filter features. Four Gabor filters are designed for facial feature extraction and the feature vector based on Gabor filters is applied to be the input of the classifier. The underlying classifier is a polynomial neural network (PNN) on a reduced feature subspace learned by PCA. The detection performances of different parts of Gabor representations have been investigated. The effectiveness of the proposed method is justified in experiments on testing a large number of images. References Adini, Y., Moses, Y., Ullman, S., 1997. Face recognition: The problem of compensating for changes in illumination direction. IEEE Trans. Pattern Anal. Machine Intell. 9 (7), 721–731. Castlman, K.R., 1996. Digital Image Processing. Prentice Hall. Chellappa, R., Wilson, C.L., Sirohey, S., 1995. Human and machine recognition of faces: A survey. Proc. IEEE 83, 705– 740. Daugman, J.G., 1988. Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression.
1649
IEEE Trans. Acoust. Speech Signal Process. 36 (7), 169– 1179. Donato, G., Bartlett, M.S., Hager, J.C., 1999. Classifying facial actions. IEEE Trans. Pattern Anal. Machine Intell. 21, 974– 989. Feraud, R., Bernier, O.J., Viallet, J.E., Collobert, M., 2001. A fast and accurate face detector based on neural networks. IEEE Trans. Pattern Anal. Machine Intell. 23 (1), 42–53. Heisele, B., Serre, T., Prentice, S., Poggio, T., 2003. Hierarchical classification and feature reduction for fast face detection with support vector machines. Pattern Recognition 36, 2007–2017. Hjelmas, R., Low, B.K., 2001. Face detection: A survey. Comput. Vision and Image Understanding 83, 236–274. Huang, L.L., Shimizu, A., Hagihara, Y., Kobatake, H., 2003. Face detection from cluttered images using a polynomial neural network. Neurocomputing 51, 197–211. Lin, C., Fan, K., 2001. Triangle-based approach to the detection of human face. Pattern Recognition 34, 1271– 1284. Liu, C., 2003. A Bayesian discriminating features methods for face detection. IEEE Trans. Pattern Anal. Machine Intell. 25, 725–740. Liu, C., Wechester, H., 2002. Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans. Image Process. 11 (4), 467–476. Osuna, E., Freund, R., Girosi, F., 1997. Training support vector machines: An application to face detection. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition. pp. 130–136. Robbins, H., Monro, S., 1951. A stochastic approximation method. Ann. Math. Stat. 22, 400–407. Rowley, H.A., Baluja, S., Kanade, T., 1998. Neural networkbased face detection. IEEE Trans. Pattern Anal. Machine Intell. 20 (1), 23–38. Schu¨rmann, J., 1996. Pattern Classification: a Unified View of Statistical Pattern Recognition and Neural Networks. Wiley Interscience. Sung, K.K., Poggio, T., 1998. Example-based learning for viewbased human face detection. IEEE Trans. Pattern Anal. Machine Intell. 20 (1), 39–50. Wong, K., Lam, K., Siu, W., 2001. An efficient algorithm for human face detection and facial feature extraction under different conditions. Pattern Recognition 34, 1993–2004. Yang, M.H., Kriegman, D.J., Ahuja, N., 2001. Face detection using multimodal density models. Comput. Vision and Image Understanding 84, 264–284. Yang, M.H., Kriegman, D.J., Ahuja, N., 2002. Detecting faces in images a survey. IEEE Trans. Pattern Anal. Machine Intell. 24 (1), 34–58. Zhang, J., Yan, Y., Lades, M., 1997. Face recognition: Eigenface, elastic matching, neural nets. Proc. IEEE 85, 1423–1435.
Pattern Recognition Letters 26 (2005) 1650–1657 www.elsevier.com/locate/patrec
Color text image binarization based on binary texture analysis Bin Wang *, Xiang-Feng Li, Feng Liu, Fu-Qiao Hu Institute of Image Processing and Pattern Recognition, Shanghai Jiaotong University, Shanghai 200030, PR China Received 10 October 2003; received in revised form 13 October 2004 Available online 14 April 2005 Communicated by E. Backer
Abstract In this paper, a novel binarization algorithm for color text images is presented. This algorithm effectively integrates color clustering and binary texture analysis, and is capable of handling situations with complex backgrounds. In this algorithm, dimensionality reduction and graph theoretical clustering are first employed. As the result, binary images related to clusters can be obtained. Binary texture analysis is then performed on each candidate binary image. Two kinds of effective texture features, run-length histogram and spatial-size distribution related respectively, are extracted and explored. Cooperating with an LDA classifier, the optimal candidate of the best binarization effect is obtained. Experiments with images collected from Internet has been carried out and compared with existing techniques, both show the effectiveness of the algorithm. 2005 Elsevier B.V. All rights reserved. Keywords: Color text image; Binarization; Binary texture features
1. Introduction Text is a kind of important feature for information retrieval applications. For these applications, as the first step, text in the image should be recognized for its semantic meanings. But current optical character recognition (OCR) technologies are
*
Corresponding author. Tel.: +86 216 236 2527; fax: +86 216 236 2529. E-mail address: [email protected] (B. Wang).
mostly restricted to recognize text against clean backgrounds. Thus binarization techniques, which aim to separate text from image backgrounds and obtain a clean representation, are usually adopted as an indispensable preprocessing. Most existing binarization techniques are thresholding related. Basically, these techniques can be categorized into two categories: global thresholding and local or adaptive thresholding. Global thresholding methods tempt to binarize the image with a single threshold. Among some most powerful global techniques, OtsuÕs algorithm can achieve
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2004.12.006
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
high performance in terms of uniformity of thresholded regions and correctness of segmentation boundaries (Sahoo et al., 1988). Liu and Srihari (1997) proposed a document binarization method in which OtsuÕs algorithm was used to obtain candidate thresholds. Then, texture features were measured from each thresholded image, based on which the best threshold were picked. In contrast to global ones, adaptive or local methods change the threshold dynamically over the image field according to some local information. Wellner (1993), an adaptive algorithm was developed by Wellner for the DigitalDesk. The method calculated threshold values at each point of estimated background illumination based on a moving average of local pixel intensities. For images with low contrast, variable background intensity and noise, local algorithms work well. However, it appeared that techniques of both above categories perform poorly under complex backgrounds. In recent years, text information extraction from images and video has attracted particular research attention and a variety of related methods have been proposed. A survey of existing methods for text detection in images and video can be found in (Chen and Luettin, 2000), and another one focusing more generally on text information extraction from images and videos was given in (Jung et al., 2004). Text detection/location results are sub-images containing only the text region in the original image, thus they are still gray-scale or color images. Generally, as mentioned above, these sub-images, especially color ones, cannot be directly fed into OCR tools for recognition unless appropriate binarization has been carried out. These images will be called text block images, or just text images in this paper, so as to distinguish them from conventional document images. Compared with conventional document images, text images possess following properties: • With only few words in an image, but these words often convey important information related to the content of the original image; • With complex backgrounds, especially for color text images; • With varying text size, fonts and colors, even for a same word.
1651
These features make no satisfactory results guaranteed when applying existing methods, which may have been well applied to conventional document images, upon these text images, especially color ones, for binarization. In this paper, a novel binarization algorithm is proposed. It is designed to overcome the limitations of existing techniques for color text images. The proposed algorithm efficiently integrates color clustering and binary texture feature analysis. Two kinds of features capable of effectively characterizing text-like binary textures, including run-length histogram and spatial-size distribution related features, are extracted and explored. In addition, in order to handle varying colors of the text in the image, a combination strategy is implemented among the binary images obtained by color clustering. The effective cooperation of these techniques enable the algorithm survives those complex background conditions, as existing techniques tend to fail.
2. Binary texture analysis Texture is the term used to characterize the contextual property of areas in images, and it is one of the main features intensively utilized in image processing and pattern recognition. Most of the known texture models are based on gray-level images, and generally, image texture is defined as a function of the spatial variation in pixel intensities, or more specifically, the spatial variation of pixel intensities which form certain repeated patterns. Texture features are numerical values calculated to describe properties of the image texture under some certain texture model. In (Mihran and Jain, 1993), texture features was categorised as statistical, geometrical, structural, model based and signal processing features in terms of the texture model employed. For general gray-level images, texture features based on statistical information of the variation of pixel intensities are most explored. As a kind of special texture type, binary or boolean texture is the texture appeared in binary images. Compared with general gray-level images, a binary image consists only two intensity levels:
1652
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
‘‘1’’ for foreground pixels, and ‘‘0’’ for background pixels. Furthermore, if only foreground (or background) regions of the binary image are of interest, as is the focus of this paper, there will be no variation of pixel intensities. Under such situation, it is apparent that general statistical models mainly focusing on pixel intensity variation are no longer suitable to characterize binary textures. For binary image textures, geometrical or shape properties and spatial distribution patterns become the most important. Thus in this paper, other than the numerous existing texture description methods based on second order statistics as given in (Mihran and Jain, 1993), two kinds of new features were taken into account in the proposed binarization method: one is based on run-length description, the other spatial-size distribution (SSD) related. As will be shown in this paper, both of these features are capable of effectively characterizing text-like textures in binary images. 2.1. Run-length based features In boolean model for texture modeling, onedimensional version is the simplest form. As texture is a two-dimensional property, directional scannings of the image are used to bypass the problem of dimensionality (Garcia-Sevilla and Petrou, 1999). One of the direct outputs of such
scannings is the run and run based binary texture characterization. A run is a maximum contiguous set of constantgray-level pixels located in a scan line, which can then be described by gray-level a, length r, and direction h, denoted as B(a, r, h). Let B(a, r) be the number of runs in all directions of length r and gray-level a. Texture description features can be obtained from computation of continuous probability of the length and gray-level of runs in the image (Sonka et al., 1999). Compared with conventional statistical methods, texture features based on run-length analysis have not attracted the same attention in image analysis applications. In this paper, study objects are binary images, and only foreground horizontal runs are of interest; thus a 1 and r 2 [1, . . . , L], where L is the maximum acceptable run-length. Then it is convenient to shorten B(a, r) to B(r). Denote B(R), R = [1, . . . , L], the one-dimensional array contains the frequencies of each run, called run-length histogram. Examples of run-length histograms for different binary images are shown in Fig. 1. Features of interest are the maximum probability and the stroke width. The maximum probability is defined as the highest frequency in the run length histogram, excluding the unit run-length, which is always related to noise. That is, MAXPROB ¼ max BðrÞ; r2R
r 6¼ 1:
ð1Þ
Fig. 1. Different binary images and their corresponding run-length histogram. (a) Normal binary image and (b) binary text image.
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
And the run-length of the highest frequency is: SW ¼ arg max BðrÞ; r2R
r 6¼ 1:
ð2Þ
It actually reflects the average stroke width of the dominant text in the image, thus in (Liu and Srihari, 1997), it was directly called stroke width feature. 2.2. Spatial-size distribution related features Binary images are the main concern of this paper. Thus, there is no variation of intensity as in general gray-level texture analysis. The most important here are the objects or components in the binary image and their spatial arrangement patterns. Same as the grayscale histogram of an image, the run-length histogram is a 1D global representation of the image. Although it contains the essential information for stroke-like features, it is still incompetent to convey the true 2D distribution patterns in the image. Thus, mathematical morphology based binary texture modeling is also studied in this paper. Mathematical morphology has been proverbially recognized as a powerful tool for binary image analysis. Under the framework of mathematical morphology, granulometries are defined as ordered sets of morphological openings or closings, each of which removes image details below a certain size (Vincent, 2000). The granulometric size distribution, or pattern spectrum, which show how the number of foreground pixels in the image changes as a function of the size parameters (Maragos, 1989), is one of the main mathematical morphology related approaches to texture description. But one of the main drawback of pattern spectrum for binary texture modeling is that it conveys only the object size distribution in the image, and very different textures may share the same pattern spectrum. This has been overcome by a new spatial size distribution (SSD) descriptor (Ayala and Domingo, 2001), through incorporating spatial distribution information into its conventional counterpart. The (p,q)-SSD of the image G with respect to a convex and compact set U containing the origin is defined as multiple integrals of the areas v(•) of
1653
granulometries in terms of spatial position variables h1, . . . ,hq (Ayala and Domingo, 2001): 1 F G;U ðk1 ; . . . ; kp ; l1 ; . . . ; lq Þ ¼ vðGÞqþ1 Z Z vðG \ G þ h1 \ . . . \ G þ hq Þ lq U
l1 U
e e e vð WðGÞ \ WðGÞ þ h1 \ \ WðGÞ þ hq Þ dh1 . . . dhq
ð3Þ
where k stands for scale of the elements and l stands for scale of the support set, and ðpÞ ð1Þ e WðGÞ ¼ Wkp Wk1 ðGÞ ; ð4Þ is the composition of the different granulometries. The elemental operation Wk(G) is defined as: Wk ðGÞ ¼ G kEð0; 1Þ ¼ G Eð0; kÞ
ð5Þ
that is, it is a morphological opening operation upon the image G with a element structure E under scale k. The joint density function associated with FG,U is: fG;U ðk1 ; . . . ; kp ; l1 ; . . . ; lq Þ ¼
opþq F G;U ðk1 ; . . . ; kp ; l! ; . . . ; lq Þ : ok1 ; . . . ; okp ol1 ; . . . ; olq
ð6Þ
For discrete sets or discrete images, such joint density function can be written as: fG;U ðk1 ; . . . ; kp ; l1 ; . . . ; lq Þ X sgnðu;vÞ ¼ ð1Þ F ðG;U Þ ðu; vÞ
ð7Þ
ðu;vÞ2N
where N ¼ fðu; vÞ : ui ¼ ki or ui ¼ ki 1; vj ¼ l or vj ¼ lj 1g and sgnðu; vÞ ¼ ]fui : ui ¼ ki 1g þ ]fvj : vj ¼ lj 1g where ]{•} stands for the counting operation, and consequently ]{ui : ui = ki 1} is the number of uis which satisfy ui = ki 1. In this paper, SSD features of order (1, 1) and (2, 1) are of interest. The value of element size k is set the same as that of the stroke-width feature
1654
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
SW of the image; the support-set size l varies from 0 to 9 in increments of 3. Thus total eight SSD features can be obtained for each binary image.
3. Color text image binarization The proposed binarization method for color text images consists of four main steps: color space dimensionality reduction, color clustering, texture feature extraction, and selection of the optimal binary image. The flowchart is shown in Fig. 2. 3.1. Dimensionality reduction and clustering Considering properties of human vision, there is large amount of redundancy in the 24-bit RGB representation of color images. Jones and Rehg (2002) reported that 77% of the possible 24-bit
Cluster 1
RGB colors were never encountered. In this paper, their research result is employed to serve as effective preprocessing. In our application, we found that representing each of the RGB channels with only 4 bits introduced little, or even no perceptible visual degradation, as shown in Fig. 3(b). Another attractive feature of this operation is its convenience, simply through performing a 4-bit rightshifting on each RGB channel. Though the dimensionality of the color space has been dramatically reduced, it is still of 16 · 16 · 16; an unsupervised graph theoretical clustering (Matas and Kittler, 1995) is employed for further information congregation. Unlike other techniques, the graph theoretical clustering does not act on image pixels directly, but on the condensed 3D color histogram. Thereby, it suffers little from variation of image sizes. Fig. 3 gives an example of the dimensionality reduction and
Texture feature extraction Texture feature extraction
Cluster 2 Color image
Color dimensionality reduction
Graph theoretical clustering
Texture feature extraction
. . . . . .
Binary image selection
Binary text image
Texture feature extraction Cluster n
Texture feature extraction
Fig. 2. Flowchart of the proposed algorithm.
Fig. 3. Color dimensionality reduction and grap-theoretical clustering. (a) Original image, (b) dimension reduction and (c) Color clustering.
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
1655
Fig. 4. The effect of the combination strategy. (a) Original image, (b) no combination and (c) with combination.
graph theoretical clustering: image (a) is the original color image of 49,789 colors; image (b), dimensionality reduction reduces the number of colors to 816; after further clustering, in image (c), only 5 colors (clusters) remain. 3.2. Binary image generation and feature extraction For each of the clusters obtained by the graphtheoretical clustering, a binary image is constructed. Additional binary images are built through combination between/among different original binary images. Before combination take effect, connected component analysis is performed on the original binary images, which can wipe off some background components from them. According to different combination strategies and different input images, we can obtain teens to tens of binary images. The problem becomes how to identify the one giving the best binarization among these candidates. To evaluate the goodness of the candidates, the aforementioned two kinds of texture features are extracted and analyzed. 3.3. Optimal binary image selection In LiuÕs method (Liu and Srihari, 1997), a simple decision tree was employed to identify the best threshold. In the proposed method, limitations of threshold based methods, including OtsuÕs and LiuÕs, are overcome through combination among clusters, instead of selecting a single cluster related binary image as the final output. Multiple steps are implemented to obtain the best binary image. First, initial binary images with small stroke-width frequencies, or the maximumprobability feature, are discarded. Thus only clusters of prominent stroke-like features remain for
further processing. But it is often that non-text binary images can possess prominent stroke-like features in terms of run-length description, and then pass the initial evaluation. This is the reason why LiuÕs method (Liu and Srihari, 1997) still suffers from complex backgrounds. In our method, SSD features of the remained images are fed into a linear discriminant analysis (LDA) classifier for further verification. Images pass the classier are combined again to generate the final result. Comparison between results with and without combination is shown in Fig. 4. It is obvious that integrating the combination strategy into the algorithm can bring more satisfactory binarization, especially for images of several text blocks with different colors.
4. Experimental results The proposed algorithm has been evaluated with a data set of 520 color images gathered from Internet. All these images contain prominent text blocks, but with varying complexity of image backgrounds and text colors. Compared with traditional document images, most of the images gathered for experiment contain only a few words. Upon these images, existing binarization methods which can perform well on conventional document images tend to fail. In the experiment, binarization results and performance of the proposed method were evaluated and compared with two existing methods, OtsuÕs method and the run-length histogram based thresholding method by Liu and Srihari (1997). Examples of the binarization results are shown in Fig. 5. As expected, OtsuÕs algorithm, Fig. 5(b), did not perform well in separating text from
1656
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
Fig. 5. Comparison among different algorithms. (a) Original image, (b) OtsuÕs algorithm, (c) LiuÕs method and (d) Proposed method.
complicated backgrounds. Method by Liu, Fig. 5(c), performed better; but same as OtsuÕs algorithm and, in its essence, being single-threshold based technique, it still suffered much from complex backgrounds. The proposed method in this paper, as shown in Fig. 5(d), however, can survive such situations and yield satisfactory results. Furthermore, in LiuÕs method, only text of uniform stroke width was considered; while in our algorithm, text with varying stroke width in a single image can be effectively handled. Apparently, only rough visual comparison above is far from sufficient to reach an objective and convincing evaluation. In the experiment, objective performance comparison among the proposed method and existing ones was carried out in two ways: one in image level, the other in word level. In the image-level evaluation strategy, criteria to accept a given binarization result include: (1) above 80% of the text in the image are correctly separated from the background, and (2) the sum of the area for non-text background regions in the result binary image is no more than that of the accepted text regions. In the word-level evaluation
scheme, recognizable words in the original image and the result binary image which have been correctly separated from the background were counted. The word-level accuracy was then defined as the rate of the number of correctly separated words to that of total recognizable words in the test image set. The final statistics for the experimental results are shown in Table 1. They are divided into two categories, corresponding to the above-mentioned two evaluation strategies: column 2 for image-level evaluation and column 3 for word-level evaluation. From the table, it can be seen that OtsuÕs Table 1 Statistics of the experimental results Scheme
Image-level evaluation
Word-level evaluation
Total Otsu algorithm LiuÕs method (Liu and Srihari, 1997) Proposed method
520 images 353 (67.9%) 376 (72.3%)
3519 words 2397 (68.1%) 2441 (69.4%)
446 (85.8%)
2850 (81%)
The data set consists of 520 color text images.
B. Wang et al. / Pattern Recognition Letters 26 (2005) 1650–1657
1657
either globally or locally. For these techniques no satisfactory binarization results are guaranteed as applied to images with complex backgrounds. In this paper, we have presented a new scheme for color text image binarization which efficiently integrates color clustering and binary texture analysis. Compared with existing ones, the proposed method is robust enough to survive those complicated situations. First results on our data set of above 500 color text images collected from Internet are very encouraging.
Fig. 6. Time consuming regarding to image size.
algorithm only gained an accuracy of 67.9% in the image-level evaluation, a better accuracy of 72.3% was obtained by LiuÕs method; superior to both the two techniques, the proposed algorithm reached an accuracy of 85.8%. In the word-level evaluation, accuracy for OtsuÕs algorithm was 68.1%, and it is 69.4% for LiuÕs method; both the existing methods were under that of the proposed one, which is 81%, with a wide gap of more than 10%. The experimental results show that, compared to existing techniques, the proposed algorithm can effectively separate text from color images with complex backgrounds, and is capable of obtaining cleaner binarization results. In order to further illustrate the efficiency of the proposed method, computational time consumption was also evaluated. Our method was implemented in C++, and had been run on a computer with 2.0GHz Pentium-4M CPU and 256M DRAM. Time consumption in terms of size of the image to process is plotted in Fig. 6. The plot shows that the time required is not changed smoothly as the image size varying. It is variation of the complexity of the images that results in the oscillation in the plot.
5. Conclusion Binarization is an important processing in computer vision, especially for document or text image related applications. Most existing techniques for document image binarization utilize thresholding,
References Ayala, G., Domingo, J., 2001. Spatial size distributions: Applications to shape and texture analysis. IEEE Trans. Pattern Anal. Machine Intell. 23 (12), 1430–1442. Chen, D.T., Luettin, J., 2000. A survey of text detection and recognition in images and videos. IDIAP-RR-00 38, IDIAP. Garcia-Sevilla, P., Petrou, M., 1999. Classification of binary textures using the one-dimensional boolean model. IEEE Trans. Image Process. 8 (10), 1457–1462. Jones, M.J., Rehg, J.M., 2002. Statistical color models with application to skin detection. Int. J. Comput. Vision 46 (1), 81–96. Jung, K., Kim, K.I., Jain, A.K., 2004. Text information extraction in images and video: A survey. Pattern Recog. 37 (5), 977–997. Liu, Y., Srihari, S.N., 1997. Document image binarization based on texture features. IEEE Trans. Pattern Anal. Machine Intell. 19 (5), 540–544. Maragos, P., 1989. Pattern spectrum and multiscale shape representation. IEEE Trans. Pattern Anal. Machine Intell. 11 (7), 701–716. Matas, J., Kittler, J., 1995. Spatial and feature space clustering: Applications in image analysis. In: Proceedings of the 6th International Conference on Computer Analysis of Images and Patterns, Prague, Czech Republic, September, pp. 162– 173. Mihran, T., Jain, A.K., 1993. Texture analysis. In: Handbook of Pattern Recognition and Computer Vision. World Scientific Publishing, pp. 235–276. Sahoo, P.K., Soltani, S., Wong, A.K.C., 1988. A survey of thresholding techniques. Comput. Vision Graph. Image Process. 41 (2), 233–260. Sonka, M., Hlavac, V., Boyle, R., 1999. Image Processing, Analysis, and Machine Vision. Brooks and Cole Publishing, Pacific Grove, CA. Vincent, L., 2000. Granulometries and opening trees. Fundamenta Informaticae 41 (1-2), 57–90, January. Wellner, P.D., 1993. Adaptive thresholding on the digitaldesk. Technical Report EPC-93-110, Rank Xerox Research Center, Cambridge Laboratory.
Pattern Recognition Letters 26 (2005) 1658–1674 www.elsevier.com/locate/patrec
Online mining maximal frequent structures in continuous landmark melody streams Hua-Fu Li a
b
a,*
, Suh-Yin Lee a, Man-Kwan Shan
b
Department of Computer Science and Information Engineering, National Chiao-Tung University, 1001 Ta Hsueh Road, Hsin-Chu 300, Taiwan Department of Computer Science, National Chengchi University, 64, Sec. 2, Zhi-nan Road, Wenshan, Taipei 116, Taiwan Received 10 January 2004; received in revised form 13 November 2004 Available online 14 April 2005 Communicated by E. Backer
Abstract In this paper, we address the problem of online mining maximal frequent structures (Type I & II melody structures) in unbounded, continuous landmark melody streams. An efficient algorithm, called MMSLMS (Maximal Melody Structures of Landmark Melody Streams), is developed for online incremental mining of maximal frequent melody substructures in one scan of the continuous melody streams. In MMSLMS, a space-efficient scheme, called CMB (Chord-set Memory Border), is proposed to constrain the upper-bound of space requirement of maximal frequent melody structures in such a streaming environment. Theoretical analysis and experimental study show that our algorithm is efficient and scalable for mining the set of all maximal melody structures in a landmark melody stream. 2005 Elsevier B.V. All rights reserved. Keywords: Machine learning; Data mining; Landmark melody stream; Maximal melody structure; Online algorithm
1. Introduction Recently, database and knowledge discovery communities have focused on a new data model, *
Corresponding author. Tel.: +886 35731901; fax: +886 35724176. E-mail addresses: hfl[email protected] (H.-F. Li), sylee@ csie.nctu.edu.tw (S.-Y. Lee), [email protected] (M.-K. Shan).
where data arrives in the form of continuous, rapid, huge, unbounded streams. It is often referred to as data streams or streaming data. Many applications generate large amount of data streams in real time, such as sensor data generated from sensor networks, transaction flows in retail chains, Web record and click streams in Web applications, performance measurement in network monitoring and traffic management, call records in telecommunications, etc. In such a data
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.016
•••
User Query Streams
User Query Processor
Music ID
Melody Sequence
Buffer
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
•••
Melody (Sequence) Streams
Music Database
1659
Melody Stream Processor Maximal Melody Structure Streams
Summary Data Structure in Main Memory
Fig. 1. Computation model for music melody streams.
stream model, knowledge discovery has two major characteristics (Babcock et al., 2002). First, the volume of a continuous stream over its lifetime could be huge and fast changing. Second, the continuous queries (not just one-shot queries) require timely answers, and the response time is short. Hence, it is not possible to store all the data in main memory or even in secondary storage. This motivates the design of in-memory summary data structure with small memory footprints that can support both one-time and continuous queries. In other words, data stream mining algorithms have to sacrifice the exactness of its analysis result by allowing some counting error. Although several techniques have been developed recently for discovering and analyzing the content of static music data (Bakhmutora et al., 1997; Hsu et al., 2001; Shan and Kuo, 2003; Yoshitaka and Ichikawa, 1999; Zhu et al., 2001), new techniques are needed to analyze and discover the content of streaming music data. Thus, this paper studies a new problem of how to discover the maximal melody structures in a continuous unbounded melody stream. The problem comes from the context of online music-downloading services (such as Kuro at www.music.com.tw), where the streams in question are streams of queries, i.e., music-downloading requests, sent to the server, and we are interested in finding the maximal melody structures requested by most customers during some period of time. With the computation model of music melody streams presented in Fig. 1, the melody stream processor and the summary data structure are two major components in the
music melody streaming environment. The user query processor receives user queries in the form of hTimestamp, Customer-ID, Music-IDi, and then transforms the queries into music data (i.e., melody sequences) in the form of hTimestamp, CustomerID, Music-ID, Melody-Sequencei by retrieving the music database. Note that a buffer can be optionally set for temporary storage of recent music melodies from the music melody streams. In this paper, we present a novel algorithm MMSLMS (Maximal Melody Structures of Landmark Melody Streams) for mining the set of all maximal melody structures in a landmark melody stream. Moreover, the music melody data and patterns are represented as sets of chord-sets (Type I Melody structures) or strings of chord-sets (Type II Melody structures). While providing a general framework of music stream mining, algorithm MMSLMS has two major features, namely one scan of music melody streams for online frequency collection, and prefix-tree-based compact pattern representation. With these two important features, MMSLMS is provided with the capability to work continuously in the unbounded streams for an arbitrary long time with bounded resources, and to quickly answer users queries at any time.
2. Preliminaries 2.1. Music terminologies In this section, we describe several features of music data used in this paper. For the basic
1660
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
terminologies on music, we refer to (Jones, 1974). A chord is a sounding combination of three or more notes at the same time. A note is a single symbol on a musical score, indicating the pitch and duration of what is to be sung and played. A chord-set is a set of chords (Shan and Kuo, 2003). Definition 1. The type I melody structure is represented as a set of chord-sets. The type II melody structure is represented as a string of chordsets. 2.2. Problem statement Let W = {i1,i2, . . . , in} be a set of chord-sets, called items for simplicity. A melody sequence S with m chord-sets is denoted by S = hx1x2 xmi, where xi 2 W, "i = 1,2, . . . , m. A block is a set of melody sequences. Definition 2. A landmark melody stream LMS = [B1,B2, . . . , BN), is an infinite sequence of blocks, where each block Bi is associated with a block identifier i, and N is the identifier of the ‘‘latest’’ block BN. The current length of LMS, written as jLMSj, is N. The blocks arrive in some order (implicitly by arrival time or explicitly by timestamp), and may be seen only once. Definition 3. A set Y W is called an item-set, i.e., a set of chord-sets. k-item-set is represented by (y1y2 yk). The support of an item-set Y, denoted as r(Y), is the number of melody sequences containing Y as a subset in the LMS seen so far. An item-set is frequent if its support is greater than or equal to minsup Æ jLMSj, where minsup is a user-specified minimum support threshold in the range of [0, 1], and jLMSj is the current length of the landmark melody stream LMS. Definition 4. A string Z is called an item-string, i.e., a string of chord-sets. A k-item-string is represented by hz1z2 zki, where zi 2 W, "i = 1,2, . . ., k. The support of an item-string Z, denoted as r(Z), is the number of melody sequences containing Z as a substring in the LMS seen so far. An item-string is frequent if its support is greater than or equal to minsup Æ jLMSj,
where minsup is a user-specified minimum support threshold in the range of [0, 1], and jLMSj is the current length of the landmark melody stream seen so far. Definition 5. A frequent item-set (or item-string) is called maximal if it is not a subset (or substring) of any other frequent item-set (or itemstring). In fact, the total number of maximal melody structures is smaller than that of frequent melody structure. Hence, the type of maximal melody structures is more suitable for the performance requirements of music stream mining. Definition 6. (Problem Definition of Online Mining Maximal Melody Structures in Continuous Landmark Melody Streams.) Given a landmark melody stream LMS = [B1,B2, . . . , BN) and the user specified minimum support, minsup, in the range of [0, 1], the problem of online mining maximal melody substructures is to discover the set of all maximal melody structures, i.e., maximal item-sets or maximal item-strings, in single one scan of the landmark music stream. 2.3. Main performance requirements of music melody stream mining The main performance challenges of mining melody streams are: (1) Online, one-pass algorithm: each sequence in the landmark melody stream is examined once. (2) Bounded-storage: limited memory for storing crucial, compressed information in summary data structure. (3) Real-time: per item processing time must be low. The proposed MMSLMS algorithm possesses all of these characteristics, while none of previously published methods (Bakhmutora et al., 1997; Hsu et al., 2001; Shan and Kuo, 2003; Yoshitaka and Ichikawa, 1999; Zhu et al., 2001) can claim the same.
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
3. Online mining maximal frequent structures in landmark melody streams 3.1. Chord-set memory border In this section, the upper bound on the number of candidate maximal melody structures is discussed, and an efficient algorithm for chord-set memory border construction is proposed. Theorem 1. Given a set of k frequent chord-sets from a landmark melody stream, an upper bound of the amount of maximal frequent melody structures is C kdk=2e . Proof. Assume that there are k frequent chordsets, i.e., k frequent items, in the current landmark melody stream. The solution space of mining all frequent item-sets in the worst case is C k1 þ C k2 þ þ C ki þ þ C kdk=2e þ þ C kk , where C k1 is the total number of frequent 1-item-sets, C ki is that of frequent i-item-sets, and C kk is that of frequent k-item-sets. We observe that the value of C kdk=2e is the maximum value among all the binominal coefficient C ki ; 8i ¼ 1; 2; . . . ; k, in mining all frequent i-item-sets. In other words, the number of frequent dk/2e-item-sets is a maximum. We will prove the number of maximal frequent itemsets can not be greater than the value C kdk=2e , i.e., C kdk=2e is the upper bound. We prove it by contradiction. Assume that the value of C kdk=2e is not the maximum number of maximal frequent item-sets, i.e., a larger upper bound U exists, where U > C kdk=2e . Consider that there are one or more frequent melody structures with length L, where L > dk/2e. If F is a frequent melody structure with length dk/2e + i and it is maximal, where i = 1,2, . . ., k dk/2e, then all of the substructures of F are frequent, which is based on the antimonotone Apriori heuristic (Agrawal and Srikant, 1994): if any i-item-set (or i-item-string) is not frequent, its (i + 1)-item-set (or (i + 1)-item-string) can never be frequent, but not maximal, which is based on the definition 5: a frequent item-set (or item-string) is called maximal if it is not a subset (or substring) of any other frequent item-set (or itemstring). In other words, it means that when one
1661
maximal frequent structure with length L, where L > dk/2e, is added, at most L frequent melody structures with length L 1, are decremented from the current collection of maximal frequent melody structures found so far. Hence, the maximum number of maximal melody structures is changed from U to U 0 , where U 0 = U + 1 L, which is not greater than C kdk=2e . This conflicts with the assumption of U > C kdk=2e and results in a contradiction. Thus the statement is proven to be true. Therefore, we conclude that the maximum number of maximal melody structures is C kdk=2e in the problem of online mining maximal melody structures in a landmark melody stream. h
Example 1. Assume that there are five frequent items (i.e., frequent 1-item-sets) a, b, c, d, and e in the landmark melody stream as shown in Fig. 2. Let MF denote the total number of maximal frequent item-sets. At this point, a, b, c, d and e are maximal and MF ¼ C 51 . Based on the Apriori heuristic, C 52 frequent 2-item-sets are discovered in the worst case. In this case, these frequent 2-item-sets are also maximal and those frequent 1-item-sets are not maximal any more. The current MF is C 51 þ C 52 C 51 ¼ C 52 . Next, C 53 frequent 3-item-sets are found in the worst case. These frequent 3-item-sets are maximal but the sub-sets of the maximal 3-item-sets, i.e., frequent 2-item-sets, are not maximal any more. Now, the MF becomes C 52 þ C 53 C 52 ¼ C 53 . At this time, suppose the frequent 4-item-set abcd exists in this instance and it is also a maximal 4-itemset. The frequent subsets, with length three, of abcd, i.e., abc, abd, acd and bcd, are not maximal any more. Now, the MF becomes C 53 þ 1 4 ¼ 7, i.e., abcd, ace, ade, bcd, bce, bde, cde are maximal frequent item-sets. The new MF is smaller than the upper bound C 5d5=2e . Hence, we can find that if one or more frequent item-sets with length L, where L > d5/2e, are added into the collection of maximal frequent item-sets found so far, the value of MF would be changed and would be less than C 5d5=2e . Consequently, the C 5d5=2e is the upper bound of the number of maximal melody structures.
1662
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
a
b
c
d
C 15
e
ab
ac
ad
ae
bc
bd
be
cd
ce
de
C 52
abc
abd
abe
acd
ace
ade
bcd
bce
bde
cde
C 53
abcd
abce
abde
acde
bcde
C 54 C 55
abcde Fig. 2. Item-set enumeration lattice with respect to five items: a, b, c, d and e.
The key property of algorithm MMSLMS is derived from the recent work (Karp et al., 2003) for finding frequent elements in streaming data. The basic scheme of mining chord-sets from music data streams is generalized from the well-known algorithm (Fisher and Salzberg, 1982) for determining whether a value (majority element) occurs more than n/2 times, i.e., minsup = 0.5, in a data stream of length n. The method can be extended to an arbitrary value of minsup. The scheme is processed as follows. At any given time, a superset of k probably frequent chord-sets with at most 1/minsup times is maintained. Initially, the set is empty. As a chord-set is read from the melody sequence in the current block, two operations are performed as follows. First, if the current chord-set is not contained in the superset and some entries are free, it is inserted into the superset with a count set to one. Second, if the chord-set is already in the superset, its count is incremented by one. However, if the superset is full, the count of each entry in the superset is decremented by one, and the chord-sets whose frequencies are just one are dropped. The method thus identifies at most k candidates for having appeared more than n/(k + 1) times, and uses O(1/minsup) memory entries.
3.2. The proposed algorithm: MMSLMS Algorithm MMSLMS has three modules: MMSLMS-buffer, MMSLMS-summary, and MMSLMS-mine. MMSLMS-buffer repeatedly reads in a block of melody sequences into available main memory. All compressed and essential information about the maximal melody structures will be maintained in the MMSLMS-summary. Finally, MMSLMS-mine finds the maximal melody structures by a depth-first manner in the current MMSLMS-summary. Therefore, the challenges of online mining landmark melody streams lie in the design of a space-efficient representation of the in-memory summary data structure and a fast discovery algorithm for finding maximal melody structures in real time. 3.2.1. MMSLMS-summary First of all, the in-memory data structure MMSLMS-summary is defined and the constructing process of MMSLMS-summary is discussed. Then we use a running example to illustrate. Definition 7. A MMSLMS-summary is an extended prefix-tree-based summary data structure defined below.
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
1. MMSLMS-summary consists of a CMB (Chordset Memory Border), and a set of MPI-trees (Maximal Prefix-Item trees of item-suffixes) denoted as MPI-trees(item-suffixes). 2. Each node in the MPI-tree(item-suffix) consists of four fields: item-id, support, block-id and node-link, where item-id is the item identifier of the inserting item, support registers the number of melody sequences represented by a portion of the path reaching the node with the item-id, the value of block-id assigned to a new node is the block identifier of the current block, and node-link links up a node with the next node with the same item-id in the same MPI-tree or null if there is none. 3. Each entry in the CMB consists of four fields: item-id, support, block-id, and head of node-link (a pointer links to the root node of the MPI-tree with the same item-id), abbreviated as head-link, where item-id registers which item identifier the entry represents, support records the number of transactions containing the item carrying the item-id, the value of block-id assigned to a new entry is the block identifier of current block, and head-link points to the root node of the MPI-tree(item-suffix). Notice that each entry with item-id in the CMB is an item-suffix and it is also the root node of the MPI-tree(item-id). 4. Each MPI-tree(item-suffix) has a specific CMBtable (Chord-set Memory Border table) with respect to the item-suffix (denoted as CMBtable(item-suffix)). The CMB-table(item-suffix) is composed of four fields, namely item-id, support, block-id, and head-link. The CMBtable(item-suffix) operates the same as the CMB except that the field head-link links to the first node carrying the item-id in the MPI-tree(itemsuffix). Notice that jCMB-table(item-suffix)j = jCMBj in the worst case, where jCMBj denotes the total number of entries in the CMB. The construction of MMSLMS-summary is described as follows. First of all, MMSLMS reads a melody sequence S from the current block. Then, MMSLMS projects the sequence S into many subsequences and inserts these subsequences into the CMB and MPI-trees. In details, each melody
1663
sequence S, such as hx1,x2, . . . , xmi, in the current block should be projected by inserting m itemsuffix melody subsequences into the MMSLMSsummary. In other words, the melody sequence S = hx1,x2, . . . , xmi is converted into m melody subsequences; that is, hx1,x2, . . . , xmi, hx2,x3, . . . , xmi, . . . , hxm1,xmi, and hxmi. The m melody subsequences are called item-suffix sequences, since the first item of each melody subsequence is an itemsuffix of the original melody sequence S. This step is called sequence projection, and is denoted as Sequence-Projection (S) = {x1jS,x2jS, . . . , xijS, . . . , xmjS}, where xijS = hxi,xi+1, . . . , xmi, "i = 1,2, . . . , m. Furthermore, the cost of sequence projection of a melody sequence with length m is (m2 + m)/ 2, i.e., m + (m 1) + + 2 + 1. After Sequence-Projection (S), MMSLMS algorithm removes the original melody sequence S from the MMSLMS-buffer. Next, the set of items in these item-suffix sequences are inserted into the CMB and the MPI-trees(item-suffixes) as a branch, and the CMB-table(item-suffixes) are updated according to the item-suffixes. If an itemset (or item-string) share a prefix with an item-set (or item-string) already in the tree, the new itemset (or item-string) will share a prefix of the branch representing that item-set (or item-string). In addition, a support counter is associated with each node in the tree. The counter is updated when an item-suffix sequence causes the insertion of a new branch. In order to limit the memory size of the summary data structure MMSLMS-summary, a space pruning technique is performed. Let the minimum support threshold be minsup, in the range of [0, 1], and the current length of the landmark melody stream be N. The rule for space pruning is as follows. A melody structure E is deleted if E.support < minsup Æ N. E is called an infrequent melody structure. After pruning all infrequent melody structures from the CMB, CMB-table-(itemsuffix) and MPI-trees, the MMSLMS-summary contains all information about frequent melody structures of the landmark melody stream generated so far. Example 2 below illustrates the algorithm step by step. Note that the h i of sequences are omitted for clear presentation.
1664
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
Example 2. Let a block Bj of the landmark melody stream LMS be hacdefi, habei, hcefi, hacdfi, hcefi and hdfi, and the minimum support threshold be 0.5 (i.e., minsup = 0.5), where a, b, c, d, e and f are chord-sets (i.e., items) in a landmark melody stream seen so far. MMSLMS algorithm constructs the MMSLMS-summary with respect to the incoming block Bj and prunes all item-sets that are infrequent from the current MMSLMS-summary in the following steps. Note that each node or entry represented as (f1:f2:f3) is composed of three fields: item-id, support, and block-id. For example, (a:2:j) indicates that, from block Bj, item a appeared twice. Step 1: MMSLMS reads current block Bj into main memory for constructing the MMSLMSsummary. (a) First melody sequence acdef: First of all, MMSLMS algorithm reads the first melody sequence acdef and calls the Sequence-Projection (acdef). Then MMSLMS inserts the item-suffix sequences acdef, cdef, def, ef, and f into
the CMB, [MPI-tree(a), CMB-table(a)], [MPI-tree(c), CMB-table(c)], [MPItree(d), CMB-table(d)], [MPI-tree(e), CMB-table(e)], and [MPI-tree(f), CMB-table(f)], respectively. The result is shown in Fig. 3. In the following sub-steps, as demonstrated in Fig. 4 through Fig. 9, the head-links of each CMB-table (item-suffix) are omitted for concise presentation. (b) Second melody sequence abe: MMSLMS reads the second melody sequence abe and calls the Sequence-Projection (abe). Next, MMSLMS inserts the item-suffix sequences abe, be and e into the CMB, [MPI-tree(a), CMB-table(a)], [MPItree(b), CMB-table(b)] and [MPItree(e), CMB-table(e)], respectively. The result is shown in Fig. 4. (c) Third melody sequence cef: MMSLMS reads the third melody sequence cef and calls the Sequence-Projection (cef). Then, MMSLMS inserts the item-suffix sequences cef, ef and f into the CMB,
Fig. 3. MMSLMS-summary construction after inserting first melody sequence acdef in block Bj. In the following sub-steps, as demonstrated in Fig. 4 through Fig. 9, the head-links of each CMB-table (item-suffix) are omitted for concise presentation.
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
1665
Fig. 4. MMSLMS-summary construction after inserting second melody sequence abe.
[MPI-tree(c), CMB-table(c)], [MPItree(e), CMB-table(e)] and [MPItree(f), CMB-table(f)], respectively. The result is shown in Fig. 5. (d) Fourth melody sequence acdf: MMSLMS reads the fourth melody sequence acdf and calls the Sequence-Projection (acdf). Next, MMSLMS inserts the item-suffix sequences acdf, cdf, df and f into the CMB, [MPI-tree(a), CMB-table(a)], [MPI-tree(c), CMBtable(c)], [MPI-tree(d), CMB-table(d)] and [MPI-tree(f), CMB-table(f)], respectively. The result is shown in Fig. 6. (e) Fifth melody sequence cef: MMSLMS reads the fifth melody sequence cef and calls the Sequence-Projection (cef). Then, MMSLMS inserts the item-suffix sequences cef, ef and f into the CMB, [MPI-tree(c), CMB-table(c)], [MPItree(e), CMB-table(e)] and [MPItree(f), CMB-table(f)], respectively. The result is shown in Fig. 7.
(f) Sixth melody sequence df: MMSLMS reads the sixth melody sequence df and calls the Sequence-Projection (df). Next, MMSLMS inserts the item-suffix sequences df and f into the CMB, [MPI-tree(d), CMB-table(d)] and [MPItree(f), CMB-table(f)], respectively. The result is shown in Fig. 8. Step 2: After computing the current block Bj, all infrequent melody structures are pruned by MMSLMS from the current MMSLMSsummary. At this time, MMSLMS deletes the MPI-tree(b) and its corresponding CMB-table(b), and prunes the entry b from the CMB, since item b is an infrequent item; that is, r(b) < minsup Æ jLMSj, where r(b) = 1 and minsup Æ jLMSj = 0.5 Æ 6 = 3. Next, MMSLMS reconstructs the MPI-tree(a) by eliminating the information about the infrequent item b. The result is shown in Fig. 9. The description stated above is the constructing process of MMSLMS-summary with respect to the
1666
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
Fig. 5. MMSLMS-summary construction after inserting third melody sequence cef.
Fig. 6. MMSLMS-summary construction after inserting third melody sequence acdf.
incoming block over a landmark melody stream. The MMSLMS-summary construction algorithm is depicted in Fig. 10.
3.2.2. MMSLMS-mine In this section, the module, called MMSLMSmine, of mining maximal melody item-sets and
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
1667
Fig. 7. MMSLMS-summary construction after inserting fifth melody sequence cef.
Fig. 8. MMSLMS-summary construction after inserting sixth melody sequence df.
maximal melody item-strings from the current MMSLMS-summary is discussed (Fig. 11).
First of all, given an entry id (from left to right, for example) in the current CMB, MMSLMS-mine
1668
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
Fig. 9. Current MMSLMS-summary after pruning all infrequent melody structures with respect to infrequent item b.
generates candidate maximal melody structures by a top-down approach. The top-down method uses the frequent items (i.e., chord-sets) of CMB-table(id) and item id to generate the candidates. The generating order of these candidates is determined by the size of item-set, from item-set size 1 + jCMB-table(id)j down to size 2. Note that, the generating order ends in 2-item-sets because all frequent entries in the current CMB-table are frequent 1-item-sets. Then MMSLMS-mine checks these candidates whether they are frequent or not by traversing the MPI-tree(id). The MPI-tree traversing principle is described as follows. First, MMSLMS-mine generates a candidate maximal melody item-set, (j + 1)-item-set, containing the item id and all items of the CMB-table(id), where jCMB-table(id)j = j. Second, MMSLMS-mine traverses the MPI-tree via the node-links of the frequent candidate. After that if the candidate is not a frequent item-set, MMSLMS-mine generates substructure candidates with j-item-sets. Next, MMSLMS-mine executes the same MPI-tree traversing scheme for item-set counting. The process stops when MMSLMS-mine finds all maximal
frequent melody structures from the current MMSLMS-summary. Moreover, MMSLMS-mine stores these maximal melody structures into a temporal pattern list, called MMSLMS-list. Notice that MMSLMS-mine can find the set of frequent 2-itemsets by combining the item-suffix id with the frequent items of the CMB-table(id). Example 3. This example illustrates the mining of the maximal melody item-sets from the current MMSLMS-summary in Fig. 9. Let the minimum support threshold be 0.5, i.e., minsup = 0.5. (1) Now, we start the maximal melody item-set mining scheme from the frequent item a. At this moment, the frequent item-set is the only 1-item-set (a), since the support of items c, d, e and f in the CMB-table (a) are less than minsup Æ jLMSj, where jLMSj = jBjj = 6. (2) Next, MMSLMS-mine starts on the second entry c for maximal melody item-set mining. MMSLMS-mine generates a candidate maximal 3-item-set (cef), and traverses the MPItree(c) for counting its support. As a result,
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
1669
Algorithm 1 (MMSLMS-summary Construction) Input: A landmark melody stream, LMS = [B1, B2, …, BN), and a user-specified minimum support threshold, minsup. Output: A current MMSLMS-summary. 1: CMB = ∅ /*initialize the CMB to empty.*/ 2: foreach block Bj do /* j = 1, 2, …, N */ 3: foreach melody sequence S = <x1, x2, …, xm> ∈ Bj (j = 1, 2, …, N) do 4: foreach item xi S do /*the CMB maintenance*/ 5: if xi ∉ CMB then 6: create a new entry (xi, 1, j, head-link) into the CMB; /* the entry form is (item-id, support, block-id, head-link)*/ 7: else /* the entry already in the CMB*/ 8: xi.support = xi.support + 1; /* increment the support of item-id xi by one*/ 9: end if 10: end for 11: call Sequence-Projection(S); /* project the sequence with every prefix-item xi for the construction of MPI-tree(xi)*/ 12: end for 13: call MMSLMS-summary-pruning(MMSLMS-summary, minsup, |LMS|); 14: end for Subroutine Sequence-Projection Input: A melody sequence S = <x1, x2, …, xm> and the current block-id j; Output: MPI-trees(xi), ∀i=1, 2, …, m; 1: foreach item xi (i =1, 2, …, m) do 2: MPI-tree-maintenance([xi|X], MPI-tree(xi), j); /* X = x1, x2, …, xm is the original melody sequence */ /* [xi|X] is an item-suffix melody sequence with item-suffix xi*/ 3: end for Subroutine MPI-tree-maintenance Input: An item-suffix melody sequence <xi, xi+1, …, xm> (i=1, 2, …, m), MPI-tree(xi) and the current block-id j; Output: The modified MPI-tree(xi), where i=1, 2, ..., m; 1: foreach item xi do /* i=1, 2, …, m */ 2: if MPI-tree has a child Y such that Y.item-id = xi.item-id then 3: Y.support = Y.support +1; /*increment Y’s support by one*/ 5: else 6: create a new node Y = (item-id, 1, j, node-link); /* initialize the Y’s support to 1, and link its parent link to MPI-tree, and its node-link linked to the nodes with same item-id via the node-link structure. */ 7: end if 8: end for Fig. 10. Algorithm of MMSLMS-summary construction.
the candidate (cef) is a maximal frequent item-set, since its support is 3, and it is not a sub-structure of any other maximal melody structures within the MMSLMS-list. Now, MMSLMS-mine stores the maximal item-set (cef) into the MMSLMS-list.
(3) MMSLMS-mine starts on the third entry d and generates a maximal frequent 2-itemset (df). We store this item-set (df) into the MMSLMS-list because it is not a sub-structure of any other maximal melody structures within the current MMSLMS-list.
1670
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
Subroutine MMSLMS-summary-pruning Input: An MMSLMS-summary, a user-specified minimum support threshold, minsup, and the current length of LMS, |LMS|; Output: An MMSLMS-summary which contains the set of all frequent melody structures. 1: foreach entry xi (i=1, 2, …, d) ∈ CMB, where d =|CMB| do
2: 3: 4: 5: 6: 7: 8:
if xi .support < minsup |LMS| then /* xi is not a frequent item */ delete those nodes (item-id = xi) via node-link structure; merge the fragmented sub-trees; /* a simple way is to reinsert or to join the remainder sub-trees into the MPI-tree*/; delete MPI-tree(xi); delete the entry xi from the CMB; end if end for Fig. 10 (continued)
Algorithm 2 (MMSLMS-mine) Input: A current MMSLMS-summary, the current length of landmark melody stream |LMS|, and a minimum support threshold minsup. Output: A temporal-pattern-list, MMSLMS-list, of maximal melody structures. 1: MMSLMS-list = ∅; 2: foreach entry e in the current CMB do 3: do generate a candidate maximal melody structure E with size |E| /* |E| = 1+|CMB-table(e) */ 4: counting E.support by traversing the MPI-tree(e); 5: if E.support minsup |LMS| then 6: if E ∉ MMSLMS-list and E is not a substructure of any other maximal frequent structures contained into the MMSLMS-list then 7: add E into the MMSLMS-list; 8: remove E’s substructures from the MMSLMS-list; 9: end if 10: else /* if E is not a frequent melody structure*/ 11: enumerate E into melody substructures with size |E| —1; 12: end if 13: until MMSLMS-mine find the set of all maximal frequent structures with respect to the item e; 14: end for Fig. 11. Algorithm of MMSLMS-mine.
(4) On the fourth entry e, since its maximal melody item-set (ef) is a sub-structure of previous maximal melody item-set (cef), MMSLMS-mine does not store it into the MMSLMS-list. (5) Finally, MMSLMS-mine computes the entry f, and generates a maximal frequent 1-itemset (f) directly, since the CMB-table(f) is
empty. MMSLMS-mine does not store it into the MMSLMS-list, because it is a substructure of a generated maximal item-set (cef). In conclusion, the Maximal Type I Melody Structures determined by algorithm MMSLMS are (a), (cef) and (df). Now, we describe the mining
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
principle of maximal melody item-strings, i.e., Maximal Type II Melody Structures, as below. MMSLMS-mine generates maximal melody itemstrings from the current MMSLMS-summary as shown in Fig. 9 by a depth-first-search (DFS) approach. Hence, the Maximal Type II Melody Structures determined by algorithm MMSLMS are hai, hci, hdi and hefi. Note that hfi is not maximal melody item-string since it is a sub-string of the existing maximal melody 2-item-string hefi. Based on the algorithm MMSLMS-mine in Fig. 10, we have the following lemma. Lemma 2. A melody structure is a maximal melody structure if and only if it is generated by algorithm MMSLMS-mine. Proof. Algorithm MMSLMS-mine is composed of two major steps: frequent melody structure selection (step 1) and maximal melody structure verification (step 2). These steps are performed in sequence. First of all, in the step of frequent melody structure selection, MMSLMS-mine finds frequent melody structure based on the Apriori property if any length i-item-set (or i-item-string) is not frequent, its length (i + 1)-item-set (or (i + 1)-itemstring) can never be frequent. That means MMSLMS-mine does not miss any frequent melody structures. Next, in step 2, MMSLMS-mine checks the frequent melody structures generated from step 1 against the maximal melody structures of the MMSLMS-list, a temporal pattern list of maximal melody structures. If this frequent melody structure is a sub-structure (i.e., sub-set or substring) of any other structures within the MMSLMS-list, then it is not a maximal melody structure according to the Definition 5; otherwise it is a candidate maximal melody structure before the next execution of step 2. Repeating step 1 and step 2, MMSLMS-mine can generate all the maximal melody structures contained in the MMSLMS-list. Hence, we have the lemma: a melody structure is a maximal melody structure if and only if it is generated by algorithm MMSLMS-mine. Space complexity analysis: The space requirement of MMSLMS-summary consists of two parts: the working space needed to create a CMB and the CMB-tables, and the storage space needed to
1671
maintain the set of MPI-trees. Assume that CMB contains k frequent chord-sets such as e1,e2, . . . , ei, . . . , ek at any time. Based on the Theorem 1, we know that there are at most C kdk=2e maximal frequent chord-sets in the landmark melody stream seen so far. If we construct the MMSLMSsummary for all these maximal frequent melody structures, the maximum height of all the MPItrees is dk/2e. There are 1 þ C k1 þ C k1 þ þ 1 2 k1 C dk=2e1 nodes in the MPI-tree(e1), where the value 1 indicates the root node e1 of the MPI-tree(e1), þ C k1 þ þ C k1 and C k1 1 2 dk=2e1 are internal and leaf nodes of the MPI-tree(e1). Moreover, there are 1 þ C k2 þ C k2 þ þ C k2 nodes in the 1 2 dk=2e1 ki ki MPI-tree(e2), . . ., 1 þ C ki 1 þ C 2 þ þ C dk=2e1 kðk1Þ nodes in the MPI-tree(ei), 1 þ C 1 nodes in the MPI-tree(ek1), and 1 (root) node in the MPItree(ek). Thus, the total number of nodes of MPItrees in the MMSLMS-summary is
þ C k1 þ þ C k1 ð1 þ C k1 1 2 dk=2e1 Þ þ C k2 þ þ C k2 þ ð1 þ C k2 1 2 dk=2e1 Þ þ ki ki þ ð1 þ C ki 1 þ C 2 þ þ C dk=2e1 Þ þ kðk1Þ
þ ð1 þ C 1
Þþ1
¼ ðC k1 þ C k1 þ C k1 þ þ C k1 0 1 2 dk=2e1 Þ þ C k2 þ C k2 þ þ C k2 þ ðC k2 0 1 2 dk=2e1 Þ þ ki ki ki þ ðC ki 0 þ C 1 þ C 2 þ þ C dk=2e1 Þ þ kðk1Þ
þ ðC 0
kðk1Þ
þ C1
Þ þ C kk 0 :
This number equals C k1 þ C k2 þ þ C kdk=2e based on Pascals Identity: let x and y be positive integers with x P y. Then C xþ1 ¼ C xy1 þ C xy . y Moreover, the worst case working space requires at most (k2 + k)/2 entries, which is based on the process of Sequence-Projection. Thus, the space requirement of MMSLMS-summary is ðk 2 þ kÞ=2 þ C k1 þ C k2 þ þ C kdk=2e . Finally, the upper bound of space requirement is O(2k). h The worst case space complexity of algorithm MMSLMS can be analyzed in terms of melody sequence size as described below. Assume that the average melody sequence size is m, the current
1672
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
length of the landmark melody stream is N, and the minimum support threshold is minsup. The space requirement of algorithm MMSLMS is composed of two parts, working space and storage space. The working space is used to store the CMB and CMB-tables and the storage space is used to store the MPI-trees. The working space requirement is m + (m 1) + (m 2) + + 1 and the storage space requirement is also about m + (m 1) + (m 2) + + 1. Hence, the space requirement of MMSLMS for inserting a melody sequence with average size m into MMSLMS-summary is 2[m + (m 1) + (m 2) + + 1] = m2 + m. Hence, the space requirement of the stream generated so far in the worst case is N Æ (m2 + m). Note that in the analysis, we assume that the sminsup Æ N is just one and therefore every item of the incoming melody sequence is a frequent item, which is the worst case. However, we know that the value of N increases as time progresses. Hence, the pruning mechanism of MMSLMS-summary is deployed to limit the memory requirement not to exceed an upper bound. From the space complexity analysis, it is not surprising to find that the space complexity grows exponentially into the number of frequent items in the CMB, as all frequent item-sets are represented in the data structure. It is also the solution space of the problem. Time complexity analysis: From the construction process of MMSLMS-summary, we can see that exactly one scan of a landmark melody stream is required. The cost (denoted by Time-cost(S)) of inserting a melody sequence S into the MMSLMSsummary by sequence projection is jSj + (jSj 1) + + 1 = (jSj2 + jSj)/2; that is O(jfreq(S)j2), where freq(S) is the set of frequent items in the melody sequence S. Note that jfreq(S)j 6 jSj, where jSj denotes the size of the melody sequence S. Because the items within the CMB are frequent items, therefore, the cost of inserting a melody sequence S can be stated in terms of the size of CMB. Time-cost(S) = O(jS 0 j2), where jS 0 j is the number of chord-sets of melody sequence S within the CMB. In the worst case, if the melody sequence S contains all the frequent items within the CMB, Time-cost(S) = O(jCMBj2).
4. Experimental results In this section, we first describe the data and experiment set-up used to evaluate the performance of the proposed algorithm, and then report our experimental results. 4.1. Synthetic data and experiment set-up To evaluate the performance of MMSLMS algorithm, two experiments are performed. The experiments were carried out on the IBM synthetic market-basket test data generator proposed by Agrawal and Srikant (1994). Two data streams, denoted by S10.I5.D1000K and S30.I15.D1000K, of size 1 million melody sequences each are studied. The first one, S10.I5.D1000K with 1 K unique items, has an average melody sequence size of 10 with average maximal potentially frequent structure size of 5. The second one, S30.I15.D1000K with 10 K unique items, has an average melody sequence size of 30 with average maximal potentially frequent structure size of 15. In all experiments, the melody sequences of each datasets are looked up in sequence to simulate the environment of a landmark melody stream. All the experiments are performed on a 1066-MHz Pentium III processor with 128 megabytes main memory, running on Microsoft Windows XP. In addition, all the programs are written in Microsoft/Visual C++ 6.0. 4.2. Experimental results In the first experiment, two primary factors, memory and execution time, are examined in the online mining of a landmark melody stream, since both should be bounded online as time advances. In Fig. 12(a), the execution time grows smoothly as the dataset size increases. This is because the average execution time of dataset S10.I5 and S30.I15 are about 12 and 25 s per block respectively, where a block is composed of 50,000 melody sequences. In other words, the computation time of dataset S10.I5 by algorithm MMSLMS is 12 s every 50,000 melody sequences, and for dataset S30.I15 is 25 s every 50,000 melody sequences. Hence, it grows smoothly as the dataset size increases. The memory usage in Fig. 12(b) for both
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
1673
Fig. 12. Required resources for synthetic datasets: (a) execution time and (b) memory.
Fig. 13. (a) Linear scalability of the data stream size and (b) relative error of mining results.
synthetic datasets is stable as time progresses, indicating the feasibility of algorithm MMSLMS. Note that the synthetic landmark melody stream is partitioned into blocks with size 50 K. In the second experiment, we investigate the scalability and relative error of algorithm MMSLMS with respect to varying minimum supports. The relative error is defined as the difference between the measured support and the actual support estimation divided by the actual support. In Fig. 13(a), the execution time grows smoothly as the dataset increases (assume minsup = 0.01%) indicating linear scalability. Fig. 13(b) shows that the relative error decreases as minisup decreases, i.e., as the size of CMB decreases. Generally, the more frequent items are
maintained in the CMB, the more accurate the mining result is.
5. Conclusions In this paper, we proposed a single-pass algorithm, MMSLMS, to discover and maintain all maximal melody structures in a landmark model that contains all the melody sequences in a data stream. In the MMSLMS algorithm, an efficient in-memory summary data structure, MMSLMS-summary, is developed to record all maximal frequent structures in the current landmark model. In addition, MMSLMS uses a space-efficient scheme, the Chord-set Memory Border (CMB), to guarantee
1674
H.-F. Li et al. / Pattern Recognition Letters 26 (2005) 1658–1674
the upper-bound of space requirements of mining maximal melody sequences in a streaming environment. Theoretical analysis and experimental results with synthetic data show that MMSLMS algorithm can meet the performance requirements of data stream mining: one-scan, bounded-space and real time. Further work includes online mining maximal melody structures in count-based and timebased sliding window that contains the most recent melody sequences in a data stream. Acknowledgements The authors thank the reviewers precious comments for improving the quality of the paper. The research is supported by National Science Council of R.O.C. under grant no. NSC93-2213-E-009-043.
References Agrawal, R., Srikant, R., 1994. Fast algorithms for mining association rules. In: Proceedings of 20th International Conference on Very Large Data Bases, pp. 487–499.
Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J., 2002. Models and issues in data stream systems. In: Proceedings of 21th ACM Symposium on Principles of Database Systems, pp. 1–16. Bakhmutora, V., Gusev, V.U., Titkova, T.N., 1997. The search for adaptations in song melodies. Computer Music Journal 21 (1), 58–67. Fisher, M.J., Salzberg, S.L., 1982. Finding a majority among n votes: solution to problem 81-5. Journal of Algorithms 3 (4), 362–380. Hsu, J.L., Liu, C.C., Chen, A.L.P., 2001. Discovering nontrivial repeating patterns in music data. IEEE Transactions on Multimedia 3 (3), 311–325. Jones, G.T., 1974. Music Theory. Harper & Row, Publishers, New York. Karp, R.M., Papadimitrious, C.H., Shanker, S., 2003. A simple algorithm for finding frequent elements in streams and bags. ACM Transactions on Database Systems 28 (1), 51– 55. Shan, M.-K., Kuo, F.-F., 2003. Music style mining and classification by melody. IEICE Transactions on Information and Systems E86-D (4), 655–659. Yoshitaka, A., Ichikawa, T., 1999. A survey on content-based retrieval for multimedia databases. IEEE Transactions on Knowledge and Data Engineering 11 (1), 81–93. Zhu, Y., Kankanhalli, M.S., Xu, C., 2001. Pitch tracking and melody slope matching for song retrieval. In: Proceedings of the Second IEEE Pacific Rim Conference on Multimedia: Advances in Multimedia Information, pp. 530–537.
Pattern Recognition Letters 26 (2005) 1675–1683 www.elsevier.com/locate/patrec
Correcting the Kullback–Leibler distance for feature selection Frans M. Coetzee
*
GenuOne, Inc., 2 Copley Square, Boston, MA 02216, USA Received 31 December 2003; received in revised form 23 October 2004 Available online 14 April 2005 Communicated by E. Backer
Abstract A frequent practice in feature selection is to maximize the Kullback–Leibler (K–L) distance between target classes. In this note we show that this common custom is frequently suboptimal, since it fails to take into account the fact that classification occurs using a finite number of samples. In classification, the variance and higher order moments of the likelihood function should be taken into account to select feature subsets, and the Kullback–Leibler distance only relates to the mean separation. We derive appropriate expressions and show that these can lead to major increases in performance. Ó 2005 Elsevier B.V. All rights reserved. Keywords: Kullback–Leibler distance; Feature selection; Classification error; Receiver operating curve (ROC); Neyman–Pearson classification
1. Introduction A common approach to feature selection is to select between feature subsets such that the Kullback–Leibler distance between the resulting class conditional probability densities is maximized. In this paper we show that this approach does not
* Address: GenuOne, Inc., ADL, 33 McComb Road, Princeton, NJ 08540, USA. Tel.: +1 609 921 3681; fax: +1 609 258 2799. E-mail address: [email protected]
properly account for the finite sample sizes that appear when classifying finite sequences of mixed samples. We present corrected criteria for evaluating feature subsets, which can lead to major performance improvements. The following notation frames the problem. We assume that we are provided groups of N < 1 samples to be classified, all taken from one of two class-dependent distributions p(xjHi), i = 0, 1 over a finite alphabet X of size M defined by a specific subset. For clarity in notation, we may as appropriate henceforth also denote p(xjH0) by q, and p(xjH1) by p, respectively.
0167-8655/$ - see front matter Ó 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.014
1676
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
The Kullback–Leibler distance is a scalar summarizing the dissimilarity of two density functions. For densities p and q it is given by Dðp : qÞ ¼
M X
pðvi Þ lnðpðvi Þ=qðvi ÞÞ
ð1Þ
i¼1
with the standard assumption of non-zero probability of all alphabet elements for the two class distributions.1 Since the K–L divergence is a single deterministic scalar that summarizes the difference of arbitrarily high-dimensional functions, it is a useful construct in algorithms. It has a number of uses in coding theory and arises most naturally in the study of frequentist approaches to probability (e.g. the method of types—Cover and Thomas, 1993). Its use in feature selection appears straightforward: when selecting between two sets of features in building a classifier, select the set on which the two classes have the most widely separated densities, as measured by the K–L divergence. In this paper we investigate this claim more carefully. For optimal classification, a classifier compares the likelihood ratio computed on the data to a threshold. It turns out that the set of outputs of any optimal classifier evaluated on input data from a class, has a mean expressed by a K–L divergence of the classconditional pdfs. However, as a random variable, the likelihood ratio also has higher order moments, which impact its being compared with the optimal threshold for class separation. This paper shows that by accounting for these higher order moments, the performance in classifying samples drawn from two sources can be much improved. This argument is developed in the next few sections; we return to further discussion in Sections 5 and 6. To formally frame the problem, let us denote two possible feature subsets by a and b, and the class conditional densities on the symbol set defined by the feature product, by pa(x) and pb(x) under hypothesis H1, and qa(x) and qb(x) under 1
Problems with zero probability of a symbol can be reduced to this type of problem, with aberrant symbols processed independently.
hypothesis H0, respectively. The common procedure is: estimate either the one-sided distance D(p:q), or more generally the symmetric K–L distance (also known as J-distance) J(p:q) = D(p:q) + D(q:p) for different subsets of features, and select the feature subset that maximizes the chosen distance. In short, the rule is: If
J ðpa : qa Þ > J ðpb : qb Þ select a; else select b ð2Þ
This approach has long been used, either explicitly, or implicitly as a building block in other approaches (see for example, summaries in Kanal, 1974; Boekee and Van Der Lubbe, 1979). The most extensive analysis of this approach was performed in the classic paper by Novovicova et al. (1996), where the authors assumed a particular form of parametric density for the classes, and the J-divergence is optimized using an embedded EM algorithm. However, even here the use of the J-divergence as being optimal for finite sample classifications is explicitly assumed, rather than proved (Eq. (12) in the reference). Our approach further differs from earlier approaches in that we directly account for the likely desired operating point of the classification step; the optimal features in a classification problem is generally a function of the desired operating point. We will defer further discussion of references until our method has been explained.
2. Classification and the K–L distance In this section we consider the relationships between the statistics of the likelihood ratio test on a source classification problem, the K–L distribution, and the error that occurs when the optimal Bayes classifier is used for finite sample classification. The classical Bayesian classification solution minimizes the error of choosing the class from which the samples were obtained, based on the class-conditional densities and the priors of the classes. For linear costs on Type I and II errors, the well known theory (Van Trees, 1971) yields a likelihood based test of the form:
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
If lðxÞ ¼ lnðpðx1 ; . . . ; xN jH 1 Þ=pðx1 ; . . . ; xN jH 0 ÞÞ > T select H 1 ; else select H 0 ð3Þ Here T is a scalar threshold determined by the cost of different errors and relative priors of the two distributions. Note that in the ideal Bayesian design case the discriminator system directly implements the above function. Since all samples in (3) are assumed to arise from one source, the likelihood ratio is a random variable that can be conditioned on the source classes. With the additional assumption of independence of samples, we find: lðxjH i Þ ¼ lnðpðx1 ; x2 ; . . . ; xN Þ=qðx1 ; x2 ; . . . ; xN ÞÞjH i N X ¼ lnðpðxj Þ=qðxj ÞÞjH i ð4Þ j¼1
The Kullback–Leibler distance arises naturally by converting the summation above to a frequency formulation (the continuous analogue being Lebesgue integration). Denoting the number of times that symbol vj occurs in the N samples by #(vj), rearranging the summation over the alphabet, and normalizing by the number of samples N, we find: " # M X lN ðxjH i Þ ¼ ð#ðvj Þ=N Þ lnðpðvj Þ=qðvj ÞÞjH i j¼1
ð5Þ Since #(vj) is a binomial variable, it follows directly that: #ðvj Þ=N jH 1 ! pðvj Þ
ð6Þ
#ðvj Þ=N jH 0 ! qðvj Þ
ð7Þ
as N ! 1 by almost any metric of convergence that is of interest (e.g. expectation, probability, mean square, and uniform all included). As a result: Dðq : pÞ; i ¼ 0 EflN ðxÞjH i g ¼ ð8Þ Dðp : qÞ; i¼1 where we ignore that, strictly speaking, equality holds only for those densities where the probability of a symbol is an integer multiple of 1/N. From this equation we see that the standard approach
1677
uses the separation of the means of the two class-conditional likelihood ratio distributions as a proxy for the degree of overlap of the same distributions. However, for finite N, the likelihood ratio lN(x) is itself a non-trivial random variable, and some metric that more accurately captures the gross outlines of its distribution should be used. For simplicity, later we will use a Gaussian approximation that yields metrics similar to the Mahalanobis distance, although other metrics could be explored. Using the constraint that the frequency counts have to lie on a simplex, the joint probability of the frequency counts is multinomial. The classconditional means are given by Eq. (8), while application of some algebra shows that the variance is given by var flN ðxÞjH 1 g ¼
1 ½Wðp : qÞ D2 ðp : qÞ N
ð9Þ
var flN ðxÞjH 0 g ¼
1 ½Wðq : pÞ D2 ðq : pÞ N
ð10Þ
where we define: Wðp : qÞ ¼
M X
pðvj Þ½lnðpðvj Þ=qðvj ÞÞ
2
ð11Þ
j¼1
We now consider the error in classification that can be expected when the likelihood ratio is used to implement the optimal Bayes classifier for finite N. While for general densities no closed form solution exists, we can make headway by assuming each class-conditioned likelihood ratio is approximately Gaussian. This assumption is usually well motivated since the linear combination of binomial distributions approaches a Gaussian distribution under rather general conditions for even moderate N (Johnson and Kotz, 1969; Feller, 1950). It is straightforward to show that given two one dimensional Gaussian distributions G(l0, r0) and G(l1, r1), where we assume means l1 > l0, variances r20 and r21 , and threshold T, the misclassification error has two major components: T l0 l1 T P ¼ pðH 0 ÞU þ pðH 1 ÞU r1 r0
ð12Þ
1678
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
pffiffiffi where U{Æ} = 1/2 erfcð= 2Þ and erfc is the standard complementary error function. Further, the ROC curve defined by false alarm rate PF{T} and detection rate PD{T} is given by T l0 P F fT g ¼ P flN ðxjH 0 Þ > T g ¼ U ð13Þ r0 T l1 P D fT g ¼ P flN ðxjH 1 Þ > T g ¼ U ð14Þ r1
Returning to the feature subset selection problem, we see that at a fixed operating point, detection will be improved by optimizing the following criterion: Xðp : q; jÞ ¼
Dðp : qÞ þ Dðq : pÞ 1=2
½Wðp : qÞ D2 ðp : qÞ 1=2 j Wðq : pÞ D2 ðq : pÞ pffiffiffiffi N Wðp : qÞ D2 ðp : qÞ
ð22Þ
We now consider two specific cases of interest. 2.1. No class preference
3. Application to feature selection
The class-conditional errors on each of the two classes are equal when the threshold: r0 r1 l0 l1 T ¼ þ ð15Þ r0 þ r1 r0 r1
From the previous section, we can now produce two modified rules for feature selection that account for the finite value of N. When Type I and II errors are to be minimized simultaneously:
whence the error is given by l l0 P ¼ U 1 ð16Þ r1 þ r0 Returning to the original problem, it follows that the limiting classification error when sample classification is performed, is given by pffiffiffiffi P ’ Uf N Cðp : qÞg ð17Þ where Cðp : qÞ ¼
Dðp : qÞ þ Dðq : pÞ ½Wðp : qÞ D ðp : qÞ1=2 þ ½Wðq : pÞ D2 ðq : pÞ1=2 2
ð18Þ
Similarly, when a fixed false alarm rate is to be achieved: If Xðpa :qa ; jÞ > Xðpb :qb ; jÞ select a; else select b ð24Þ The careful reader will require that one issue still be addressed: namely that these rules are not coincidental with Eq. (2). In short, we have to address whether the numerator and denominator in Eq. (18) are subject to an implicit dependency such that if: ð25Þ
then it follows that:
In this approach the false alarm rate PF{T} is maintained at constant level. Define j = U1{PF{Æ}}. It follows that: T ¼ l0 þ r0 j
ð19Þ
pffiffiffiffi P D fP F fgg ’ Uf N Xðp : q; jÞg
ð20Þ
where ðl1 l0 Þ r0 j r1
ð23Þ
J ðpa : qa Þ > J ðpb : qb Þ
2.2. Class preference: Neyman–Pearson classification
Xðp : q; jÞ ¼
If Cðpa : qa Þ > Cðpb : qb Þ select a; else select b
ð21Þ
Cðpa : qa Þ > Cðpb : qb Þ
ð26Þ
and similarly, for Eq. (24). We basically have to prove the rules are not simply the same in disguise. We do not expect them to always disagree, but there should be cases where they disagree with each other. This independence is easily proved by example; such an example is discussed in the next section. We note this independence is true in general: the constraints implied by Eqs. (25) and (26) are regular and impose 2 constraints on a
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
set of dimension 2(M 1) spanned by the simplexes from which the class densities are drawn. Hence neither density pairs for which the conditions hold, nor those for which they do not hold, can be considered exceptional. Combined with the fact that there is always at least one multidimensional density that exists for a set of marginal distributions, we can expect the disagreement of the selection rules for a large set of distributions.
4. Examples In this section we present results on a reference data set that show the effect of using the appropriate selection rule for classification problems. The agaricus-lepiota or ‘‘mushroom’’ dataset from the UCI Machine Learning repository (Blake and Merz, 1998) is widely used for evaluating discrete feature selection approaches. The dataset has a large number of data points (8124), and 22 features (numbered 1 through 22), all of which are discrete, with each feature having a distinct vocabulary of up to 12 symbols. These features are shown in Table 1. The rules generated on this Table 1 Features of mushroom data set, numbered 1 through 22 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Cap-shape Cap-surface Cap-color Bruises? Odor Gill-attachment Gill-spacing Gill-size Gill-color Stalk-shape Stalk-root Stalk-surface-above-ring Stalk-surface-below-ring Stalk-color-above-ring Stalk-color-below-ring Veil-type Veil-color Ring-number Ring-type Spore-print-color Population Habitat
1679
database are widely used in practice to distinguish edible and poisonous mushrooms—if you buy wild mushrooms, your continued health may in fact depend on the accuracy of this database. The large amount of data relative to feature space allows for accurate evaluations of performance, and bounds on performance for different feature sets and rules are known (Duch et al., 1997). The data was used to construct histograms for subsets of groups of three features, for which the histograms are highly accurate. An extensive search over all pairs of groups of three features requires significant computation but is possible. We studied randomly selected triplets of features, where the new criteria disagreed with the standard J-divergence when selecting the optimal feature set, until we had a 1000 cases. In each case we calculated the full ROC curve on each feature subset and evaluated the optimal classifiers in full. For clarity and due to the limited space here, we only consider feature subsets where both of the new criteria differed from the J-divergence (i.e. the new criteria indicate that both at the Bayesian operating point, and the Pf = 0.15 point, the newly selected feature subset would outperform the commonly selected subset). This rather stringent joint condition occurred for roughly 10% of the feature subset pairings. Table 2 contains 10 examples of feature triplets, the values of the different selection criteria calculated for the class-conditional distributions on the subset, and the Bayesian error rate P* (equal class errors) as well as the Neyman–Pearson detection error rate P 0:15 at a false alarm rate of d Pf = 0.15 (the full ROC curve was calculated and interpolated to find these error rates). Table 3 contains 10 examples of feature triplets where the new criteria yielded a different subset selection from the common procedure, but the performance was in fact worse. We ascribe this error to the fact that the new criteria models the likelihood statistics only up to second order; while usually this yields improvement over just using the mean separation (J-divergence), it is not always adequate. We note that such inferior performance is not the norm: Table 4 shows that in most cases (80%+) the new criteria result in an improvement, and on average the improvement in
1680
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
Table 2 Selected triplets of features and performance, where selection criteria differ, and new criteria yield an improvement Set a
Set b
Ja
Jb
Ca
Cb
Xa
Xb
P a
P b
P 0:15 a
P b0:15
(2, 15, 16) (11, 13, 16) (6, 11, 19) (7, 13, 18) (10, 12, 21) (4, 6, 15) (2, 4, 12) (7, 13, 22) (13, 17, 18) (13, 16, 19)
(4, 6, 8) (8, 12, 17) (4, 11, 18) (11, 16, 22) (8, 10, 21) (7, 16, 20) (16, 18, 20) (16, 18, 20) (10, 12, 13) (4, 17, 21)
3.18 5.01 6.61 5.63 6.83 5.52 5.61 6.44 5.11 5.89
2.29 4.85 6.11 4.98 5.84 4.06 5.43 5.43 4.09 5.45
0.70 0.88 1.15 0.95 1.12 0.89 0.99 1.18 0.96 1.07
0.86 1.21 1.41 1.00 1.33 1.38 1.55 1.55 0.99 1.25
1.32 1.53 2.21 1.37 2.05 1.59 1.88 2.05 1.22 1.62
1.63 1.90 3.76 1.76 3.90 2.35 3.00 3.00 1.41 3.54
73.6 83 87.5 74.2 80.9 77.2 84.2 86.6 71.7 83.4
78.1 90.8 92.4 83.4 89.6 88.7 89.6 89.6 78.2 87.3
59.4 79.9 88.1 67.9 76.7 69.7 84.2 87.6 64.5 82.9
67.5 92.8 93.6 82.6 93.3 89.3 90.3 90.3 74.6 89
Table 3 Selected triplets of features and performance, where selection criteria differ, and new criteria cause under-performance (fewer cases overall) Set a
Set b
Ja
Jb
Ca
Cb
Xa
Xb
P a
P b
P 0:15 a
P b0:15
(10, 18, 22) (1, 14, 16) (10, 12, 17) (8, 10, 11) (9, 10, 16) (4, 10, 11) (3, 4, 16) (6, 8, 22) (6, 7, 9) (10, 18, 19)
(7, 10, 21) (11, 16, 18) (6, 13, 17) (4, 18, 20) (3, 11, 16) (4, 8, 11) (2, 8, 21) (12, 13, 16) (3, 11, 16) (6, 11, 18)
4.13 2.83 2.83 8.01 6.43 8.38 4.95 3.15 5.80 3.74
4.10 2.42 2.03 5.48 5.04 7.28 3.67 2.64 5.04 3.45
0.91 0.65 0.72 1.45 1.06 1.63 0.86 0.78 1.12 0.79
0.96 0.68 0.73 1.59 1.17 1.68 0.87 0.86 1.17 0.80
1.42 1.20 0.97 2.62 1.82 2.89 1.52 1.25 1.63 1.10
3.96 2.04 1.16 2.86 3.75 3.97 2.03 1.39 3.75 1.78
81.5 71.7 70 92 85.7 92.5 79 77.5 85.2 79.4
76.4 70.3 69.4 90.2 83.6 91.6 75.7 75.3 83.6 72.9
80.5 59 62.7 93.3 85.7 93.9 68.7 74.2 85.2 70.1
57.8 56 61.7 91.4 80.7 93.2 68.5 71.7 80.7 60.6
Table 4 Mean improvement in error on sets where new criteria are used and performance is increased, and percentage where criteria differ, and new criteria cause under-performance Case
% of sets
DP (%)
DP 0:15 (%)
Superior Inferior Overall
81.9 18.1 100
6.34 2.24 4.79
10.2 5.23 7.37
performance at both operating points is higher than when the new criteria result in inferior performance. Fig. 1 shows the receiver operating curves for two representative feature subsets a = (2, 15, 16) and b = (4, 6, 8). This case is the first in Table 2. 5. Feature space search Up to now the paper discussed only the problem of choosing between two subsets of features
for later use in a classification scheme. The general feature selection problem involves absolute, or more generally a co-final directed set ranking (Munkres, 1975) of all 2N subsets of a set of N features. Such ranking of the subsets requires, first, a method for traversing the feature subset space, and second, a local method for comparing selected pairs of subsets. Our approach addresses the second part of the problem—how to better rank any two subsets; the global search is still responsible for correctly interpreting the pairwise preference, and being robust against errors. For any reasonable sizes of the alphabet, the selection problem is computationally insoluble; in addition to the sheer computational complexity, the ranking is further dependent on the desired operating point—a point frequently unacknowledged. Even in our selected examples in Table 2, different operating points favor different features. True feature selection requires the ordering of
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
Fig. 1. Receiver operating curves (ROC), ROC1 for features a = (2, 15, 16) and ROC2 for features b = (4, 6, 8) on the mushroom data set. The new criteria yields improved performance. The operating points for an optimal Bayesian classifier minimizing Type I and II error simultaneously (straight lines), and for a Neyman–Pearson classifier with Pf = 0.15 (dashed lines) are shown.
ROC curves, not points. The brute force solution requires calculating the ROC curves for all feature subsets, and sorting based on performance at the desired operating point. And if that is not enough of a challenge, in wrapper approaches the additional load of a restricted classifier architecture is also imposed. The plethora of feature selection procedures all try to reduce the complexities involved (see Blum and Langley, 1997 for a review). Some taxonomies for grouping methods exist; the major separation is that of approaches that explicitly perform selection before classification from those where the feature selection is implicit. Explicit approaches include methods that compare performance of a seed set and the set formed by adding features to (forward approach) or trimming features from (backward approach) the seed set, often with some probabilistic or backtracking annealing components (Koller and Sahami, 1996; Pudil et al., 1994a). Our new criteria are obvious candidates for replacing the J-divergence or KL-divergence
1681
in these approaches. In other applications the architecture, estimation and evaluation are intermingled—for example in Attribute Value Taxonomy generation and tree building, split variable selection corresponds to maximizing class divergence (Baker and McCallum, 1998; Kang et al., 2004). Properly accounting for the separation of finite runlength samples is relevant and our new criteria should be considered. A further issue is performing feature selection with limited data. Little effort has been expended in this area. We note that the author previously proposed using the directed-set structure that exist on the ROC curves to bound the feature subset search to only a statistically valid region of the feature space supported by the finite data (Coetzee et al., 2001). This latter approach also sorts ROC curves, and selects families of subsets based on operating point. The approach was required in our applications where the operating point varies in a wide range. The approach used in the current paper is extremely helpful in pre-sorting large data sets for such computationally complex searches, since we can select for particular operating points, yet both criteria (23) and (24) use scalar statistics that retain the benefit of having a single number that captures performance on the dataset.
6. Conclusion It is helpful here to discuss why the effects described in this communication are not widely recognized, especially when the standard J-divergence based approaches are so widespread. It appears that a remarkable degree of confusion exists as to the number N in the likelihood ratio. In introductions of the K–L distance such as in Cover (Cover and Thomas, 1993), the problem that is analyzed (correctly) is how many samples are required to be drawn consecutively from a single distribution to distinguish between two possible source distributions. As N becomes large, equi-partition theorems hold, and the K–L distribution appears naturally in various error exponents. However, in a practical classification problem, samples are obtained from two distributions and
1682
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683
the problem is more accurately represented by a mixture distribution. Practitioners frequently confuse the two problems. Even if a large number N 0 of samples are obtained in the latter problem, what is of importance for the classification error bound is the run-length N of batches of samples guaranteed to be pulled sequentially from one of the classes—which is typically small or one in most applications. The large-number asymptotics captured by the K–L distributions do not apply simply because you have a large number of combined samples from the two distributions. Further, this effect of finite run-length is not the same as that of estimating the K–L divergence from a finite set of labeled data. The latter is also a difficult problem, but is a secondary effect relative to the focus of this paper. The other frequent source of confusion results from using K–L divergence implicitly as part of a larger feature selection approach that aims to retain information. In these approaches features are eliminated such that the per-class densities are disturbed minimally by the features being eliminated (Pudil et al., 1994b; Koller and Sahami, 1996). While intuitively these approaches should yield reasonable results, there is no reason to believe that minimizing distortion of per-class distributions as measured by K–L divergence generally minimizes subsequent classification error between the resulting class-conditional distributions. Similarly, ranking bounds on classification performance on feature subsets does not necessarily translate into selecting the optimal feature subset, especially since bounds such as Bhattacharya bounds pay a heavy penalty in discrimination and continuity to achieve their general applicability. We note that if the features of the samples are not independent, our approach still functions perfectly. The important independence requirement relates to the drawing of data samples. If the samples are drawn dependently, converting the likelihood ratio to a frequentist formulation (the Lebesgue integration in (5)), is not always possible, and the K–L distribution does not naturally arise. Little work exists on analyzing the dependent case, beyond some work on higher order extensions of the method of ‘‘types’’—notably by Csiszar (see Csiszar, 1998 for a review). The independence con-
dition is however not an issue in most problems typically considered as classification problems; the dependence problem usually crops up in areas of stochastic game theory and time series prediction. We close with one final interesting point: the selection of subsets does not change with N when neither class is favored (Eq. (18)). In contrast, when the false alarm rate has to be minimized, the run-length may influence the choice of feature subsets (Eq. (22)). In both cases, though, the overall rate of error decreases with run-length (Eqs. (17) and (20)), as one would expect.
Acknowledgment The author thanks the editor and two anonymous reviewers for suggestions that markedly clarified and improved the paper.
References Baker, L.D., McCallum, A.K., 1998. Distributional clustering of words for text classification. In: Proceedings of SIGIR98, 21st ACM International Conference on Research and Development in Information Retrieval, Melbourne, AU, pp. 96–103. Blake, C., Merz, C., 1998. UCI repository of machine learning databases. URL http://www.ics.uci.edu/ ~mlearn/MLRepository.html. Blum, A., Langley, P., 1997. Selection of relevant features and examples in machine learning. Artif. Intell. 97 (1–2), 245– 271. Boekee, D.E., Van Der Lubbe, J.C.A., 1979. Some aspects of error bounds in feature selection. Pattern Recognit. 11, 353– 360. Coetzee, F., Glover, E., Lawrence, S., Giles, C.L., 2001. Feature selection in web applications using roc inflections and power set pruning. In: Symposium on Applications and the Internet-SAINT 2001, San Diego, CA. Cover, Thomas, 1993. Principles of Information Theory. Wiley and Sons. Csiszar, I., 1998. The method of types. IEEE Trans. Inf. Theory 44 (6), 2502–2523. Duch, W., Adamczak, R., Graßbczewski, K., Ishikawa, M., Ueda, H., 1997. Extraction of crisp logical rules using constrained backpropagation networks—comparison of two new approaches. In: Proceedings of the European Symposium on Artificial Neural Networks (ESANN97), pp. 109–114.
F.M. Coetzee / Pattern Recognition Letters 26 (2005) 1675–1683 Feller, W., 1950. An Introduction to Probability Theory and Its Applications, third ed.Probability and mathematical statistics, vol. 1 John Wiley and Sons, ISBN 0-471-257087. Johnson, N.L., Kotz, S., 1969. Discrete Distributions. Probability and Mathematical Statistics. John Wiley and Sons, ISBN 0-471-44360-3. Kanal, L.N., 1974. Patterns in pattern recognition. IEEE Trans. Inf. Theory 20, 697–722. Kang, D., Silvescu, A., Zhang, J., Honavar, V., 2004. Generation of attribute value taxonomies from data for datadriven construction of accurate and compact classifiers. In: Proceedings of the Fourth IEEE International Conference on Data Mining (ICDM 04), Brighton, UK, in print. Koller, D., Sahami, M., 1996. Toward optimal feature selection. In: Proceedings of the 13th International Con-
1683
ference on Machine Learning, Morgan Kaufmann, pp. 284–292. Munkres, J.R., 1975. Topology: a First Course. Prentice-Hall, ISBN 0-13-925495-1. Novovicova, J., Pudil, P., Kittler, J., 1996. Divergence based feature selection for multimodal class densities. IEEE Trans. Pattern Anal. Mach. Intell. 18 (2), 218–223. Pudil, P., Novovicova, J., Kittler, J., 1994a. Floating search methods in feature-selection. Pattern Recognit. Lett. 15 (11), 1119–1125. Pudil, P., Novovicova, J., Kittler, J., 1994b. Simultaneous learning of decision rules and important attributes for classification problems in image analysis. Image Vis. Comput. 12 (3), 193–198. Van Trees, H.L., 1971. Detection Estimation and Modulation Theory, vols. 1–3. Wiley and Sons.
Pattern Recognition Letters 26 (2005) 1684–1690 www.elsevier.com/locate/patrec
Recursive computation method for fast encoding of vector quantization based on 2-pixel-merging sum pyramid data structure Zhibin Pan b
a,*
, Koji Kotani b, Tadahiro Ohmi
a
a New Industry Creation Hatchery Center, Tohoku University, Aza-aoba 10, Aramaki, Aoba-ku, Sendai 980-8579, Japan Department of Electronic Engineering, Graduate School of Engineering, Tohoku University, Aza-aoba 10, Aramaki, Aoba-ku, Sendai 980-8579, Japan
Received 16 February 2004; received in revised form 21 October 2004 Available online 7 April 2005 Communicated by E. Backer
Abstract Vector quantization (VQ) is a popular signal compression method. In the framework of VQ, fast search method is one of the key issues because it is the time bottleneck for VQ applications. In order to speed up VQ encoding process, how to construct some lower dimensional feature vectors for a k-dimensional original vector so as to measure the distortion between any vectors lightly becomes important. To reduce the dimension for approximately representing a kdimensional vector, the multi-resolution concept is a natural consideration. By introducing a pyramid data structure, the multi-resolution concept used in fast VQ encoding includes two aspects, which are (1) a multi-resolution distortion check method and (2) a multi-resolution distortion computation method. Some fast search methods that are based on a 4-pixel-merging (4-PM) mean pyramid data structure [Lin, S.J., Chung, K.L., Chang, L.C. 2001. An improved search algorithm for vector quantization using mean pyramid structure. Pattern Recognition Lett. 22 (3/4) 373] and a 2-pixel-merging (2-PM) sum pyramid data structure [Pan, Z., Kotani, K., Ohmi, T. 2004. An improved fast encoding algorithm for vector quantization using 2-pixel-merging sum pyramid data structure. Pattern Recognition Lett. 25 (3) 459] have already been proposed. Both of them realized the multi-resolution concept by using a multi-resolution distortion check method. However, both of them ignored the multi-resolution distortion computation method, which can also be guaranteed by the multi-resolution concept if a recursive computation way is introduced. In principle, a multi-resolution distortion computation method can completely reuse the obtained computation result that is already executed at a lower resolution level so that no waste to it will occur at all. This paper aims at
*
Corresponding author. Tel.: +81 22 217 3981; fax: +81 22 217 3986. E-mail addresses: pzb@fff.niche.tohoku.ac.jp, [email protected] (Z. Pan).
0167-8655/$ - see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.011
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
1685
improving the search efficiency of the previous work [Pan, Z., Kotani, K., Ohmi, T. 2004. An improved fast encoding algorithm for vector quantization using 2-pixel-merging sum pyramid data structure. Pattern Recognition Lett. 25 (3) 459] further by introducing a multi-resolution distortion computation method into multi-resolution distortion check method so that about half of its computational cost can be reduced mathematically. Experimental results confirmed the proposed method outperforms the previous works obviously. 2005 Elsevier B.V. All rights reserved. Keywords: Recursive computation; Fast search; Vector quantization; 2-pixel-merging; Sum pyramid
1. Introduction Vector quantization (VQ) is a widely used asymmetric signal compression method (Nasarabadi and King, 1988). The encoding process of VQ is computationally very heavy. In a conventional VQ method, an N · N input image is firstly divided into a series of non-overlapping smaller n · n image blocks. Then VQ encoding is implemented block by block sequentially. The distortion between an input image block and a codeword can be measured by squared Euclidean distance for simplicity as d 2 ðI; C i Þ ¼
k X 2 ðI j C i;j Þ
i ¼ 1; 2; . . . ; N c
ð1Þ
j¼1
where k (=n · n) is the vector dimension, I = [I1, I2, . . . , Ik] is the current image block, Ci = [Ci,1,Ci,2, . . . ,Ci,k] is the ith codeword in the codebook C = {Ciji = 1,2, . . . ,Nc}, j represents the jth element of a vector, and Nc is the codebook size. Then a best-matched codeword with minimum distortion, which is called winner afterwards, can be determined straightforwardly by d 2 ðI; C w Þ ¼ minbd 2 ðI; C i Þc C i 2C
i ¼ 1; 2; . . . ; N c
ð2Þ
where Cw means the winner and the subscript ‘‘w’’ is the winner index. This process for finding the winner is called full search (FS) because the matching is executed over the whole codebook. FS is very heavy computationally due to it performing Nc times k-dimensional Euclidean distance computation. Once ‘‘w’’ has been found, which conventionally uses much less bits than Cw, VQ only transmits this index ‘‘w’’ instead of the winner Cw itself to reduce the amount of image data in
order to realize image compression. Because the same codebook has also been stored at the receiver in VQ system, by using the received winner index ‘‘w’’ to retrieve the corresponding Cw at the receiver for an input block, it is very easy to reconstruct an image by pasting the codeword Cw one by one. Because a high dimension k of vector is the main problem for computing Euclidean distance, it is very important to use lower dimensional feature vectors to approximately express a k-dimensional original vector to roughly measure the distortion between two vectors of I and Ci so as to reject obviously unlikely codewords. There exist many fast search methods for VQ. A class of them is based on the multi-resolution concept by constructing an appropriate pyramid data structure (Lee and Chen, 1995; Lin et al., 2001; Song and Ra, 2002; Pan et al., 2004) to roughly describe the original k-dimensional vector with a series of lower dimensional feature vectors or levels in a pyramid. All of these previous works adopted a multi-resolution distortion check method, which refers to that a series of rejection checks are conducted from top level towards bottom level one by one until a rejection decision is made or the bottom level is achieved. They are very searchefficient. However, all of these previous works ignored a multi-resolution computation method, which implies that the distortion at a higher resolution level can be computed by completely reusing the obtained distortion at a previous lower resolution level and then just adding some improvements to enhance the resolution instead of computing it from the very beginning once again. As a result, there is no waste to the distortion obtained at a lower resolution level occurs. This paper aims at improving the method proposed in (Pan et al.,
1686
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
2004) by introducing a recursive computation method so as to reduce about half of its computational cost further. 2. Previous work A 4-pixel-merging (4-PM) mean pyramid data structure for fast VQ encoding as shown in Fig. 1(a) is proposed in (Burt and Adelson, 1983; Lee and Chen, 1995; Lin et al., 2001) to realize a ðu1 þ 1Þ ¼ log4 (n · n) + 1 resolution description for a k-dimensional vector, where the ‘‘Averaging’’ is implemented by a simple arithmetic average operation. Instead of using this 4-PM mean pyramid, a 2-pixel-merging (2-PM) sum pyramid as shown in Fig. 1(b) is proposed in (Pan et al., 2004), where the ‘‘Summing’’ is a summation operation. A 2-PM sum pyramid can double the resolutions of a 4-PM mean pyramid as (u2 + 1) = log2(n · n) + 1 = 2 · u1 + 1. It has already been experimentally confirmed in (Pan et al., 2004) that it is more search efficient to use a 2-PM sum pyramid than to use a 4-PM mean pyramid for fast VQ encoding in the case of not using a recursive computation method. Then, a new hierarchical rejection rule (Pan et al., 2004) based on a multi-resolution distortion check method can be obtained as
L0
d 2s;u2 ðI; C i Þ P P 2ðu2vÞ d 2s;v ðI; C i Þ P P 2ðu21Þ d 2s;1 ðI; C i Þ P 2u2 d 2s;0 ðI; C i Þ
ð3Þ
The distortion at the vthPlevel for v 2 [0 u2] 2v 2 is defined as d 2s;v ðI; C i Þ ¼ m¼1 ðSIv;m SCi;v;m Þ , where SIv,m is the mth pixel (sum) at the vth level in a 2-PM sum pyramid for I and SCi,v,m implies the same thing to Ci for m 2 [1 2v]. The real Euclidean distance d 2 ðI; C i Þ ¼ d 2s;u2 (I, Ci) holds. For a 4 · 4 block, u2 = 4. Then, at any vth level for v 2 [0 u2], if 2ðu2vÞ d 2s;v ðI; C i Þ > d 2min holds, the current real Euclidean distance d 2s;u2 ðI; C i Þ will be definitely larger than d 2min . As a result, the search can be terminated at this vth level and Ci can be rejected safely to save computational cost. Therefore, how to efficiently compute the value of distortion d 2s;v ðI; C i Þ becomes an important issue. Obviously, the search process based on Eq. (3) only used the multi-resolution distortion check method but completely ignored the multi-resolution distortion computation method. It is a waste to computation. This paper aims at introducing the multi-resolution distortion computation method into Eq. (3) in order to fully exploit the potential power of the two aspects included in the multi-resolution concept. How to reuse the
L0 (top level)
Distortion check order
(top level)
Distortion check order
L1
L1
L2
Averaging
L3 L2
n×n (a)
Summing
L4
Lu1 (bottom level)
n×n
Lu2 (bottom level)
(b)
Fig. 1. For an n · n (k = n · n) block: (a) a 4-PM mean pyramid with bottom level u1 = log4(n · n) and (b) a 2-PM sum pyramid with bottom level u2 = log2(n · n) = 2 · u1. One new level is sandwiched in-between every two levels of a conventional 4-PM pyramid (shadowed).
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
obtained d 2s;v ðI; C i Þ to compute d 2s;vþ1 ðI; C i Þ for next distortion check recursively is the key of this paper. 3. Proposed method During a winner search process, in order to avoid discarding the obtained d 2s;v ðI; C i Þ value, a recursive computation method for next d 2s;vþ1 ðI; C i Þ can be realized based on the multiresolution concept, which implies that the distortion at a higher resolution level can be computed by reusing the distortion at a lower resolution level. Based on the 2-PM sum pyramid data structure shown in Fig. 1(b), it is clear SIv,m = SIv+1,2m1 + SIv+1,2m and SCi,v,m = SCi,v+1,2m1 + SCi,v+1,2m are true for m 2 [1 2v]. Therefore, a recursive relation can be derived by rewriting d 2s;vþ1 ðI; C i Þ in the odd terms, even terms first and then using the identical equality (a2 + b2) = (a + b)2 2ab as Def d 2s;vþ1 ðI; C i Þ ¼
¼
2vþ1 X
2
ðSIvþ1;m SCi;vþ1;m Þ
m¼1 2v X
½ðSIvþ1;2m1 SCi;vþ1;2m1 Þ
2
m¼1 2
þ ðSIvþ1;2m SCi;vþ1;2m Þ 2v X ½ðSIvþ1;2m1 SCi;vþ1;2m1 Þ ¼ m¼1
þ ðSIvþ1;2m SCi;vþ1;2m Þ2 2v X 2 ½ðSIvþ1;2m1 SCi;vþ1;2m1 Þ m¼1
ðSIvþ1;2m SCi;vþ1;2m Þ 2v X 2 ½ðSIv;m Þ ðSCi;v;m Þ ¼ m¼1
2
2v X ½ðSIvþ1;2m1 SCi;vþ1;2m1 Þ m¼1
ðSIvþ1;2m SCi;vþ1;2m Þ 2v X 2 ½ð2 SIvþ1;2m1 ¼ d s;v ðI; C i Þ
1687
Eq. (4) is the core of this paper. From Eq. (4), it is obvious that the obtained d 2s;v ðI; C i Þ is completely reused again for computing d 2s;vþ1 ðI; C i Þ in a recursive way. Eq. (4) is actually a realization of the multi-resolution distortion computation method. An analysis on the computational complexity of Eq. (4) is given as follows. When d 2s;vþ1 ðI; C i ) is computed directly, it needs 2v+1 + (2v+1 1) = 2 · 2v+1 1 addition (±) and 2v+1 multiplication (·) operations. However, by storing the doubled odd terms at each level (i.e. to store 2 · SIv+1,2m1 and 2 · SCi,v+1,2m1 as a whole instead of the raw SIv+1,2m1 and SCi,v+1,2m1 for v 2 [1 u2]), Eq. (4) only needs [1 + 2 · 2v + (2v 1)] = (3/2) · 2v+1 addition (±) and 2v = (1/2) · 2v+1 multiplication (·) operations. Mathematically, addition (±) operations can be reduced to [(3/2) · 2v+1]/[2 · 2v+1 1] 3/4 and multiplication (·) operations can be reduced to [(1/2) · 2v+1]/[2v+1] = 1/2 by introducing Eq. (4) for each distortion computation at the (v + 1)th level for v 2 [0 u2 1]. Because a multiplication operation (·) is much heavier than an addition (±) operation, it is clear that Eq. (4) can save about half of the computational burden compared to a direct computation way of d 2s;vþ1 ðI; C i Þ. In principle, 2-PM sum pyramid is an extension of 4-PM mean pyramid. However, in order to realize a multi-resolution distortion computation method recursively to reduce more computational cost, only 2-PM sum pyramid can be used but 4-PM mean pyramid cannot. The reasons is that by using the formula a2 + b2 = (a + b)2 2ab, one multiplication (·) can be saved when the value of (a + b)2 is known in a 2-pixel-merging pyramid structure. In contrast, the formula a2 + b2 + c 2 + d 2 = (a + b + c + d) 2 2ab 2ac 2ad 2bc 2bd 2cd cannot save any computational cost even though the value of (a + b + c + d)2 is known in a 4-pixel-merging pyramid structure. Therefore, 2-PM sum pyramid is a more promising data structure than a 4-PM mean pyramid for fast VQ encoding. Based on the discussions above, a search flow can be summarized as follows:
m¼1
2SCi;vþ1;2m1 ÞðSIvþ1;2m SCi;vþ1;2m Þ ð4Þ
(1) Construct an accompanying 2-PM sum pyramid for each Ci based on Fig. 1(b) off-line.
1688
(2)
(3)
(4)
(5)
(6)
(7)
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
Store the doubled values of odd terms for v 2 [1 u2] levels instead of their raw values so as to take Eq. (4) into account. Sort all codewords by the real sum at L0 level in ascending order and then rearrange them along this order off-line. For an input I, construct its accompanying 2-PM sum pyramid based on Fig. 1(b) on-line. Similarly, store the doubled values of odd terms for v 2 [1 u2] levels. Find an initial nearest neighbor (NN) codeword CN among the sorted codewords by a binary search, which is the closest codeword in terms of real sum difference ds,0(I, CN) = jSI0,1 SCN,0,1j being minimum. It needs log2(Nc) times comparison (cmp) operations. Then compute and store ‘‘so far’’ d 2min ¼ d 2 ðI; C N Þ, and all d 2v;min ¼ 2u2v d 2min in order to simplify a future distortion check at the vth level for v 2 [0 u2 1]. This step needs (2 · k 1) additions (±) and ½k þ log2 ðn nÞ multiplications (·). Continue the winner search up and down around CN one by one. Once ðSI0;1 2 SCi;0;1 Þ P d 20;min holds, terminate search for the upper part of sorted codebook when i < N or the lower part when i > N; If winner search in both upper and lower directions has been terminated, search is complete. Clearly, the current ‘‘so far’’ bestmatched codeword must be the winner. Output the winner index. Then search flow returns to Step 3 for encoding another new input. Otherwise, check whether d 2s;vþ1 ðI; C i Þ P d 2vþ1;min is true or not for v 2 [0 u2 1]. If any distortion check is true, reject Ci safely (Note: Eq. (4) must be introduced here for computing d 2s;vþ1 ðI; C i Þ in a recursive computation way). If all distortion checks fail for a rejection, it implies that current Ci is a better-matched codeword, then update d 2min by d 2s;u2 ðI; C i Þ and all d 2v;min for v 2 [0 u2 1]. Meanwhile, update the winner index found ‘‘so far’’. Then, return to Step 5 to check next codeword.
4. Experimental results Simulation experiments with MATLAB are conducted. Codebooks are generated by using the 512 · 512 8-bit Lena image as a training set. Since it is mathematically very clear how much computational cost can be reduced via introducing a multiresolution distortion computation method Eq. (4) into Eq. (3), only a practical 4 · 4 block size is selected in the experiment of this paper. The reason for this selection is that a 2 · 2 block size results in a very low compression ratio and a 8 · 8 block size results in a rather low PSNR so that they are seldom to be adopted in practical VQ encoding. Because only 2-PM sum pyramid data structure but 4-PM mean pyramid data structure can use the multi-resolution distortion computation method, a performance comparison between the fast encoding method with or without using Eq. (4) is made. The search efficiency is evaluated by the total computational cost in the number of addition (±), multiplication (·) and comparison (cmp) operations per input vector, which consists of (1) finding the initial best-matched codeword CN and computing the initial d 2min ; d 2v;min for v 2 [0 u2 1]; (2) computing the distortion at each level in pyramids for a possible rejection based on Eq. (3) (non-recursive way) or Eq. (3) combined with Eq. (4) (recursive way); (3) updating the ‘‘so far’’ d 2min ; d 2v;min for v 2 [0 u2 1] if all rejection checks fail ultimately. The results are shown in Table 1. In Table 1, ‘‘Add’’ means the addition operation, ‘‘Mul’’ means the multiplication operation and ‘‘Cmp’’ means the comparison operation. From Table 1, it is also obvious that the total computational cost by introducing Eq. (4) into Eq. (3) is much less compared to using Eq. (3) only. Especially the number of multiplication operations can be reduced significantly, which benefits from computing d 2s;vþ1 ðI; C i Þ by using Eq. (4) for v 2 [0 u2 1] in a recursive way. But it cannot achieve a 50% reduction ratio because multiplication operations for computing the initial d 2min ðI; C N Þ, for generating or updating d 2v;min for v 2 [0 u2 1] and for computing d 2s;0 ðI; C i Þ are fixed. In addition, the comparison operations cannot be reduced because the number of distortion checks by using Eq. (4) does not change at all.
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
1689
Table 1 Comparison of total computational cost per input vector in the arithmetical operations for 2-PM sum pyramid data structure with full search (FS) as a baseline Codebook
Method
Operation
Lena
F-16
Pepper
Baboon
128
Full search
Add Mul Cmp Add Mul Cmp
3968 2048 128 171.16 113.81 29.13
3968 2048 128 136.46 91.23 24.78
3968 2048 128 176.87 117.06 30.89
3968 2048 128 404.36 253.33 72.25
Eqs. (3) and (4) (recursive)
Add Mul Cmp
148.34 77.62 29.13
119.73 64.12 24.78
153.45 79.48 30.89
347.20 156.43 72.25
Full search
Add Mul Cmp
7936 4096 256
7936 4096 256
7936 4096 256
7936 4096 256
Eq. (3) only (non-recursive)
Add Mul Cmp
252.98 169.05 46.43
201.78 135.08 39.78
272.09 180.17 51.02
705.39 439.77 131.26
Eqs. (3) and (4) (recursive)
Add Mul Cmp
218.18 112.08 46.43
175.83 91.53 39.78
234.79 118.43 51.02
604.44 265.16 131.26
Full search
Add Mul Cmp
15,872 8192 512
15,872 8192 512
15,872 8192 512
15,872 8192 512
Eq. (3) only (non-recursive)
Add Mul Cmp
363.53 243.26 73.98
313.50 209.70 67.43
427.89 281.19 87.55
1303.10 804.56 249.76
Eqs. (3) and (4) (recursive)
Add Mul Cmp
314.22 158.79 73.98
273.00 138.17 67.43
369.67 180.25 87.55
1116.50 475.49 249.76
Full search
Add Mul Cmp
31,744 16,384 1024
31,744 16,384 1024
31,744 16,384 1024
31,744 16,384 1024
Eq. (3) only (non-recursive)
Add Mul Cmp
492.05 334.36 112.72
478.77 322.16 114.30
657.85 434.96 149.68
2127.22 1324.27 447.91
Eqs. (3) and (4) (recursive)
Add Mul Cmp
427.80 219.44 112.72
418.78 210.42 114.30
571.47 277.76 149.68
1836.61 785.73 447.91
Eq. (3) only (non-recursive)
256
512
1024
5. Conclusion In this paper, three contributions are made. First, the multi-resolution concept that is realized by a 2-PM sum pyramid data structure in VQ
encoding is classified as two aspects, which are a multi-resolution distortion check method and a multi-resolution distortion computation method conducted in a recursive way. Second, it is mathematically made clear that only a 2-PM sum
1690
Z. Pan et al. / Pattern Recognition Letters 26 (2005) 1684–1690
pyramid data structure but a 4-PM mean pyramid data structure can benefit from this recursive computation method. Third, by completely reusing the obtain distortion d 2s;v ðI; C i Þ to compute d 2s;vþ1 ðI; C i Þ for next distortion check, it becomes unnecessary to discard anything so as to avoid the waste of computation. Experimental results confirmed that the proposed method outperforms the previous work (Pan et al., 2004) obviously. References Burt, P.J., Adelson, E., 1983. The Laplacian pyramid as a compact image code. IEEE Commun. 31 (4), 532–540.
Lee, C.H., Chen, L.H., 1995. A fast search algorithm for vector quantization using mean pyramids of codewords. IEEE Trans. Commun. 43 (2/3/4), 1697–1702. Lin, S.J., Chung, K.L., Chang, L.C., 2001. An improved search algorithm for vector quantization using mean pyramid structure. Pattern Recognition Lett. 22 (3/4), 373– 379. Nasarabadi, N.M., King, R.A., 1988. Image coding using vector quantization: A review. IEEE Trans. Commun. 36 (8), 957–971. Pan, Z., Kotani, K., Ohmi, T., 2004. An improved fast encoding algorithm for vector quantization using 2-pixelmerging sum pyramid data structure. Pattern Recognition Lett. 25 (3), 459–468. Song, B.C., Ra, J.B., 2002. A fast search algorithm for vector quantization using L2-norm pyramid of codewords. IEEE Trans. Image Process. 11 (1), 10–15.
Pattern Recognition Letters 26 (2005) 1691–1700 www.elsevier.com/locate/patrec
Design and implementation of a multi-PNN structure for discriminating one-month abstinent heroin addicts from healthy controls using the P600 component of ERP signals Ioannis Kalatzis a, Nikolaos Piliouras a, Eric Ventouras a, Charalabos C. Papageorgiou b, Ioannis A. Liappas b, Chrysoula C. Nikolaou b, Andreas D. Rabavilas b, Dionisis D. Cavouras a,* a
Department of Medical Instrumentation Technology, Technological Educational Institution of Athens, Ag. Spyridonos Street, Egaleo GR-122 10, Athens, Greece b Psychophysiology Laboratory, Eginition Hospital, Department of Psychiatry, Medical School, University of Athens, Greece Received 16 February 2004; received in revised form 5 October 2004 Available online 7 April 2005 Communicated by E. Backer
Abstract A multi-probabilistic neural network (multi-PNN) classification structure has been designed for distinguishing onemonth abstinent heroin addicts from normal controls by means of the Event-Related PotentialsÕ P600 component, selected at 15 scalp leads, elicited under a Working Memory (WM) test. The multi-PNN structure consisted of 15 optimally designed PNN lead-classifiers feeding an end-stage PNN classifier. The multi-PNN structure classified correctly all subjects. When leads were grouped into compartments, highest accuracies were achieved at the frontal (91.7%) and left temporo-central region (86.1%). Highest single-lead precision (86.1%) was found at the P3, C5 and F3 leads. These findings indicate that cognitive function, as represented by P600 during a WM task and explored by the PNN signal processing techniques, may be involved in short-term abstinent heroin addicts. Additionally, these findings indicate that these techniques may significantly facilitate computer-aided analysis of ERPs. Ó 2005 Elsevier B.V. All rights reserved. Keywords: Heroin addicts; Event-related potentials (ERPs); P600 component; Pattern recognition
*
Corresponding author. Tel.: +30 210 5385 375; fax: +30 210 5910 975. E-mail address: [email protected] (D.D. Cavouras).
0167-8655/$ - see front matter Ó 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2005.01.012
1692
I. Kalatzis et al. / Pattern Recognition Letters 26 (2005) 1691–1700
1. Introduction Event-related potentials (ERPs) are electrical potentials, usually measured on the scalp, and are distinguished for their high temporal resolution, allowing for real-time and non-invasive observation of electrical activity changes in the brain during the processing of information related to the presentation of stimuli (or events) (Fabiani et al., 2000). Late positive components of the ERP waveform, such as the P300 and the P600 components have attracted special attention in ERP research. Both components are related to working memory (WM) processes, i.e. keeping information actively in mind, the P300 being more related to the on-line updating of working memory and/or attentional operations involved in this function (Polich, 1998), while the P600, elicited between 500 and 800 ms after stimulus presentation, has been linked to hippocampal function (Guillem et al., 1998; Grunwald et al., 1999), having much in common with WM operation (Garcia-Larrea and Cezanne-Bert, 1998; Guillem et al., 1999; Frisch et al., 2003). P600 is thought to reflect the response selection stage of information processing (Falkenstein et al., 1994), i.e. the stage that Ôassigns a specific response to a specific stimulusÕ. The relationship between substance dependence, such as cocaine and/or heroin abuse, and neurophysiological functions has been previously addressed by various workers, using the P300 (Easton and Bauer, 1997; Martin and Siddle, 2003; Papageorgiou et al., 2003; Kouri et al., 1996; Bauer, 1997, 2002; Biggins et al., 1997; Attou et al., 2001) and, to a lesser extent to the P600 (Papageorgiou et al., 2001) concerning six-month, i.e. long-term, abstinent heroin addicts. As far as the application of P600 component of ERPs in picking up relevant aspects of addiction, in association with neuropsychological operation, Papageorgiou et al. (2001) provided evidence indicating that abstinent heroin addicts manifest abnormal aspects of second-pass parsing processes, as reflected by the P600 latencies, elicited during a WM test. The aim of the present study is twofold: first, to search deeper into the P600 signals by extracting new P600-signal characteristics and by employing
powerful classification procedures, to develop a pattern recognition system for discriminating drug-abusers from controls. The P600 component has only been previously employed (Vasios et al., 2002) for computer-based discrimination of normal controls from patients suffering from schizophrenia. Second, to design a novel classification system, according to which composite information is collected from all fifteen leads simultaneously and is fed into a multi-classifier structure to achieve highest classification accuracies.
2. Material and methods 2.1. Subjects Sixteen one-month abstinent heroin-abusers (4 females and 12 males), matched on age and educational level to 20 normal controls (5 females and 15 males), were examined. The former were recruited from the outpatient university clinic of Eginition Hospital of Athens, Greece. Drug abstinence was verified by urine tests. The addicts were mainly long users of heroin, they had not made prolonged use of other drugs, and had no history of mental retardation. The controls were recruited from hospital staff and local volunteer groups. All participants had no history of any neurological or hearing problems and were right-handed as assessed by the Edinburgh Inventory Test (Oldfield, 1971). Written informed consent was obtained from both patients and control subjects. 2.2. ERP generation procedure All subjects were evaluated by a computerized version of the digit span subtest of the Wechsler Adult Intelligence Scale (Wechsler, 1955). The examination procedure followed for each subject is detailed in a previous work by members of our research team (Papageorgiou et al., 2003; Papageorgiou and Rabavilas, 2003). ERPs were recorded using Ag/AgCl electrodes (leads), during the 1 s interval between the warning stimulus and the first administered number and were digitized at a sampling rate of 500 Hz. EEG activity was recorded from 15 scalp leads based on the Inter-
I. Kalatzis et al. / Pattern Recognition Letters 26 (2005) 1691–1700
national 10–20 system of Electroencephalography (Jasper, 1958), referred to both earlobes (leads at Fp1, Fp2, F3, F4, C3, C4, C5, C6, P3, P4, O1, O2, Pz, Cz, and Fz) (see Fig. 4). 2.3. Conventional statistical analysis To investigate whether the two groups of subjects could be discriminated by conventional statistical analysis methods, a step-wise discriminant method was employed, utilizing the amplitudes (parameter AMP) and latencies (parameter LAT) of the P600 component at all 15 leads. It should be noted that the equality of the covariance matrices of the variables entered for the two groups was ascertained with BoxÕs M-test. 2.4. Pattern recognition methods 2.4.1. Feature generation Nineteen features related to the P600 component (500–800 ms time interval) were extracted from each ERP-signal at each lead by means of a dedicated computer software developed in C++ for the purposes of the present study. The description and relations of the features are given in Table 1. All features were normalized to zero mean and unit standard deviation (Theodoridis and Koutroumbas, 1998), according to relation: xi l x0i ¼ ð1Þ r where xi and x0i are the ith feature values before and after the normalization respectively, and l and r are the mean value and standard deviation respectively of feature x over all subjects (addicts and normal controls). 2.4.2. The PNN classifier The probabilistic neural networks (PNNs) (Specht, 1990) are implemented by a feed-forward and one-pass structure and encapsulate the BayesÕ decision rule together with the use of Parzen estimators of dataÕs probability distribution function (PDF). The PNN classifier was chosen due to its non-parametric nature and because its training is easy and instantaneous (Specht, 1990), especially in comparison with the back-propagation neural
1693
network and the support vector machine classifiers. The discriminant equation of a PNN (equipped with the widely used Gaussian weighting function) for class k is given by the following relation, as described in (Tsai, 2000; Hajmeer and Basheer, 2002): gk ðxÞ ¼
1
d=2 Qd ð2pÞ j¼1 rj
" 2 # Nk d 1 X 1X xj xij exp N k i¼1 2 j¼1 rj
ð2Þ
where x = [x1x2 xd]T is the test pattern vector to be classified, xi is the ith training pattern vector, Nk is the number of patterns in class k, rj are the standard deviations of the distributions of the pattern vector element variables, and d is the feature space dimensionality. The test pattern x is classified to the class with the larger discriminant function value. 2.4.3. Best feature selection procedure To optimize classification performance, the best feature combination had to be determined at each one of the 15 leads, giving the highest classdiscrimination with the least number of features. This was accomplished by employing the exhaustive search method (Theodoridis and Koutroumbas, 1998) of all possible 2, 3, 4, 5, and 6 feature combinations. Combinations with higher numbers of features were also tested, employing the forward stepwise feature selection technique (Theodoridis and Koutroumbas, 1998) and the PNN classifier. The best-feature combinations thus determined were used to design at each lead the PNN classifier, using the leave-one-out method for discriminating heroin addicts from normal controls. 2.4.4. Compartmental classification Neural activity leading to the production of P600 signals has been associated with different brain structures such as the frontal, temporal and parietal regions, which participate in information processing during recognition memory (Papageorgiou et al., 2003). A careful observation of the lead placements on the scalp in Fig. 4 will
1694
I. Kalatzis et al. / Pattern Recognition Letters 26 (2005) 1691–1700
Table 1 Signal waveform feature descriptions and definitions s/n
Feature
Description
Definition
1. 2. 3.
Time interval to maximum signal value Maximum signal value LAT/AMP ratio
tsmax ¼ ft j sðtÞ ¼ smax g smax = max{s(t)} LAR ¼ tsmax =smax
The absolute value of AMP
AAMP = jsmaxj
The absolute value of LAR
ALAR ¼ jtsmax =smaxj
6.
Latency (LAT) Amplitude (AMP) Latency/Amplitude ratio (LAR) Absolute Amplitude (AAMP) Absolute latency/ amplitude ratio (ALAR) Positive area (PAR)
The sum of the positive signal values
Ap ¼
7.
Negative area (NAR)
The sum of the negative signal values
An ¼
8.
Absolute negative area (ANAR) Total area (TAR) Absolute total area (ATAR) Total absolute area (TAAR) Average absolute signal slope (TAAS)
The absolute value of NAR
ANAR = jAnj
The sum of all signal values
TAR = Apn = Ap + An ATAR = jApnj
The sum of absolute signal values
Apjnj = Apos + jAnegj
13.
Peak-to-peak (PP)
The difference between maximum and minimum signal values
14.
Peak-to-peak time window (PPT)
15.
Peak-to-peak slope (PPS)
16.
Z