ML之NB：利用朴素贝叶斯NB算法(TfidfVectorizer+不去除停用词)对20类新闻文本数据集进行分类预测、评估

ML之NB：利用朴素贝叶斯NB算法(TfidfVectorizer+不去除停用词)对20类新闻文本数据集进行分类预测、评估

目录

输出结果

设计思路

核心代码

输出结果

设计思路

核心代码

  

   

    

     

    
    

     

      class TfidfVectorizer Found at: sklearn.feature_extraction.text
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

      class TfidfVectorizer(CountVectorizer):
     
    
   

    

     

    
    

     
 """Convert a collection of raw documents to a matrix of TF-IDF features.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Equivalent to CountVectorizer followed by TfidfTransformer.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Read more in the :ref:`User Guide <text_feature_extraction>`.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Parameters
     
    
   

    

     

    
    

     

       ----------
     
    
   

    

     

    
    

     

       input : string {'filename', 'file', 'content'}
     
    
   

    

     

    
    

     

       If 'filename', the sequence passed as an argument to fit is
     
    
   

    

     

    
    

     

       expected to be a list of filenames that need reading to fetch
     
    
   

    

     

    
    

     

       the raw content to analyze.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       If 'file', the sequence items must have a 'read' method (file-like
     
    
   

    

     

    
    

     

       object) that is called to fetch the bytes in memory.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Otherwise the input is expected to be the sequence strings or
     
    
   

    

     

    
    

     

       bytes items are expected to be analyzed directly.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       encoding : string, 'utf-8' by default.
     
    
   

    

     

    
    

     

       If bytes or files are given to analyze, this encoding is used to
     
    
   

    

     

    
    

     

       decode.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       decode_error : {'strict', 'ignore', 'replace'}
     
    
   

    

     

    
    

     

       Instruction on what to do if a byte sequence is given to analyze that
     
    
   

    

     

    
    

     

       contains characters not of the given `encoding`. By default, it is
     
    
   

    

     

    
    

     

       'strict', meaning that a UnicodeDecodeError will be raised. Other
     
    
   

    

     

    
    

     

       values are 'ignore' and 'replace'.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       strip_accents : {'ascii', 'unicode', None}
     
    
   

    

     

    
    

     

       Remove accents during the preprocessing step.
     
    
   

    

     

    
    

     

       'ascii' is a fast method that only works on characters that have
     
    
   

    

     

    
    

     

       an direct ASCII mapping.
     
    
   

    

     

    
    

     

       'unicode' is a slightly slower method that works on any characters.
     
    
   

    

     

    
    

     

       None (default) does nothing.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       analyzer : string, {'word', 'char'} or callable
     
    
   

    

     

    
    

     

       Whether the feature should be made of word or character n-grams.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       If a callable is passed it is used to extract the sequence of features
     
    
   

    

     

    
    

     

       out of the raw, unprocessed input.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       preprocessor : callable or None (default)
     
    
   

    

     

    
    

     

       Override the preprocessing (string transformation) stage while
     
    
   

    

     

    
    

     

       preserving the tokenizing and n-grams generation steps.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       tokenizer : callable or None (default)
     
    
   

    

     

    
    

     

       Override the string tokenization step while preserving the
     
    
   

    

     

    
    

     

       preprocessing and n-grams generation steps.
     
    
   

    

     

    
    

     

       Only applies if ``analyzer == 'word'``.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       ngram_range : tuple (min_n, max_n)
     
    
   

    

     

    
    

     

       The lower and upper boundary of the range of n-values for different
     
    
   

    

     

    
    

     

       n-grams to be extracted. All values of n such that min_n <= n <= max_n
     
    
   

    

     

    
    

     

       will be used.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       stop_words : string {'english'}, list, or None (default)
     
    
   

    

     

    
    

     

       If a string, it is passed to _check_stop_list and the appropriate stop
     
    
   

    

     

    
    

     

       list is returned. 'english' is currently the only supported string
     
    
   

    

     

    
    

     

       value.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       If a list, that list is assumed to contain stop words, all of which
     
    
   

    

     

    
    

     

       will be removed from the resulting tokens.
     
    
   

    

     

    
    

     

       Only applies if ``analyzer == 'word'``.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       If None, no stop words will be used. max_df can be set to a value
     
    
   

    

     

    
    

     

       in the range [0.7, 1.0) to automatically detect and filter stop
     
    
   

    

     

    
    

     

       words based on intra corpus document frequency of terms.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       lowercase : boolean, default True
     
    
   

    

     

    
    

     

       Convert all characters to lowercase before tokenizing.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       token_pattern : string
     
    
   

    

     

    
    

     

       Regular expression denoting what constitutes a "token", only used
     
    
   

    

     

    
    

     

       if ``analyzer == 'word'``. The default regexp selects tokens of 2
     
    
   

    

     

    
    

     

       or more alphanumeric characters (punctuation is completely ignored
     
    
   

    

     

    
    

     

       and always treated as a token separator).
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       max_df : float in range [0.0, 1.0] or int, default=1.0
     
    
   

    

     

    
    

     

       When building the vocabulary ignore terms that have a document
     
    
   

    

     

    
    

     

       frequency strictly higher than the given threshold (corpus-specific
     
    
   

    

     

    
    

     

       stop words).
     
    
   

    

     

    
    

     

       If float, the parameter represents a proportion of documents, integer
     
    
   

    

     

    
    

     

       absolute counts.
     
    
   

    

     

    
    

     

       This parameter is ignored if vocabulary is not None.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       min_df : float in range [0.0, 1.0] or int, default=1
     
    
   

    

     

    
    

     

       When building the vocabulary ignore terms that have a document
     
    
   

    

     

    
    

     

       frequency strictly lower than the given threshold. This value is also
     
    
   

    

     

    
    

     

       called cut-off in the literature.
     
    
   

    

     

    
    

     

       If float, the parameter represents a proportion of documents, integer
     
    
   

    

     

    
    

     

       absolute counts.
     
    
   

    

     

    
    

     

       This parameter is ignored if vocabulary is not None.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       max_features : int or None, default=None
     
    
   

    

     

    
    

     

       If not None, build a vocabulary that only consider the top
     
    
   

    

     

    
    

     

       max_features ordered by term frequency across the corpus.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       This parameter is ignored if vocabulary is not None.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       vocabulary : Mapping or iterable, optional
     
    
   

    

     

    
    

     

       Either a Mapping (e.g., a dict) where keys are terms and values are
     
    
   

    

     

    
    

     

       indices in the feature matrix, or an iterable over terms. If not
     
    
   

    

     

    
    

     

       given, a vocabulary is determined from the input documents.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       binary : boolean, default=False
     
    
   

    

     

    
    

     

       If True, all non-zero term counts are set to 1. This does not mean
     
    
   

    

     

    
    

     

       outputs will have only 0/1 values, only that the tf term in tf-idf
     
    
   

    

     

    
    

     

       is binary. (Set idf and normalization to False to get 0/1 outputs.)
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       dtype : type, optional
     
    
   

    

     

    
    

     

       Type of the matrix returned by fit_transform() or transform().
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       norm : 'l1', 'l2' or None, optional
     
    
   

    

     

    
    

     

       Norm used to normalize term vectors. None for no normalization.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       use_idf : boolean, default=True
     
    
   

    

     

    
    

     

       Enable inverse-document-frequency reweighting.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       smooth_idf : boolean, default=True
     
    
   

    

     

    
    

     

       Smooth idf weights by adding one to document frequencies, as if an
     
    
   

    

     

    
    

     

       extra document was seen containing every term in the collection
     
    
   

    

     

    
    

     

       exactly once. Prevents zero divisions.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       sublinear_tf : boolean, default=False
     
    
   

    

     

    
    

     

       Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Attributes
     
    
   

    

     

    
    

     

       ----------
     
    
   

    

     

    
    

     

       vocabulary_ : dict
     
    
   

    

     

    
    

     

       A mapping of terms to feature indices.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       idf_ : array, shape = [n_features], or None
     
    
   

    

     

    
    

     

       The learned idf vector (global term weights)
     
    
   

    

     

    
    

     

       when ``use_idf`` is set to True, None otherwise.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       stop_words_ : set
     
    
   

    

     

    
    

     

       Terms that were ignored because they either:
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       - occurred in too many documents (`max_df`)
     
    
   

    

     

    
    

     

       - occurred in too few documents (`min_df`)
     
    
   

    

     

    
    

     

       - were cut off by feature selection (`max_features`).
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       This is only available if no vocabulary was given.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       See also
     
    
   

    

     

    
    

     

       --------
     
    
   

    

     

    
    

     

       CountVectorizer
     
    
   

    

     

    
    

     

       Tokenize the documents and count the occurrences of token and 
     
    
   

    

     

    
    

     

       return
     
    
   

    

     

    
    

     

       them as a sparse matrix
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       TfidfTransformer
     
    
   

    

     

    
    

     

       Apply Term Frequency Inverse Document Frequency normalization to a
     
    
   

    

     

    
    

     

       sparse matrix of occurrence counts.
     
    
   

    

     

    
    

     

       
     
    
   

    

     

    
    

     

       Notes
     
    
   

    

     

    
    

     

       -----
     
    
   

    

     

    
    

     

       The ``stop_words_`` attribute can get large and increase the model size
     
    
   

    

     

    
    

     

       when pickling. This attribute is provided only for introspection and can
     
    
   

    

     

    
    

     

       be safely removed using delattr or set to None before pickling.
     
    
   

    

     

    
    

     

       """
     
    
   

    

     

    
    

     
 def __init__(self, input='content', encoding='utf-8', 
     
    
   

    

     

    
    

     

       decode_error='strict', strip_accents=None, lowercase=True, 
     
    
   

    

     

    
    

     

       preprocessor=None, tokenizer=None, analyzer='word', 
     
    
   

    

     

    
    

     

       stop_words=None, token_pattern=r"(?u)\b\w\w+\b", 
     
    
   

    

     

    
    

     

       ngram_range=(1, 1), max_df=1.0, min_df=1, 
     
    
   

    

     

    
    

     

       max_features=None, vocabulary=None, binary=False, 
     
    
   

    

     

    
    

     

       dtype=np.int64, norm='l2', use_idf=True, smooth_idf=True, 
     
    
   

    

     

    
    

     

       sublinear_tf=False):
     
    
   

    

     

    
    

     

       super(TfidfVectorizer, self).__init__(input=input, encoding=encoding, 
     
    
   

    

     

    
    

     

       decode_error=decode_error, strip_accents=strip_accents, 
     
    
   

    

     

    
    

     

       lowercase=lowercase, preprocessor=preprocessor, tokenizer=tokenizer, 
     
    
   

    

     

    
    

     

       analyzer=analyzer, stop_words=stop_words, 
     
    
   

    

     

    
    

     

       token_pattern=token_pattern, ngram_range=ngram_range, 
     
    
   

    

     

    
    

     

       max_df=max_df, min_df=min_df, max_features=max_features, 
     
    
   

    

     

    
    

     

       vocabulary=vocabulary, binary=binary, dtype=dtype)
     
    
   

    

     

    
    

     

       self._tfidf = TfidfTransformer(norm=norm, use_idf=use_idf, 
     
    
   

    

     

    
    

     

       smooth_idf=smooth_idf, 
     
    
   

    

     

    
    

     

       sublinear_tf=sublinear_tf)
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     
 # Broadcast the TF-IDF parameters to the underlying transformer 
     
    
   

    

     

    
    

     

       instance
     
    
   

    

     

    
    

     
 # for easy grid search and repr
     
    
   

    

     

    
    

     

       @property
     
    
   

    

     

    
    

     
 def norm(self):
     
    
   

    

     

    
    

     
 return self._tfidf.norm
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @norm.setter
     
    
   

    

     

    
    

     
 def norm(self, value):
     
    
   

    

     

    
    

     

       self._tfidf.norm = value
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @property
     
    
   

    

     

    
    

     
 def use_idf(self):
     
    
   

    

     

    
    

     
 return self._tfidf.use_idf
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @use_idf.setter
     
    
   

    

     

    
    

     
 def use_idf(self, value):
     
    
   

    

     

    
    

     

       self._tfidf.use_idf = value
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @property
     
    
   

    

     

    
    

     
 def smooth_idf(self):
     
    
   

    

     

    
    

     
 return self._tfidf.smooth_idf
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @smooth_idf.setter
     
    
   

    

     

    
    

     
 def smooth_idf(self, value):
     
    
   

    

     

    
    

     

       self._tfidf.smooth_idf = value
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @property
     
    
   

    

     

    
    

     
 def sublinear_tf(self):
     
    
   

    

     

    
    

     
 return self._tfidf.sublinear_tf
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @sublinear_tf.setter
     
    
   

    

     

    
    

     
 def sublinear_tf(self, value):
     
    
   

    

     

    
    

     

       self._tfidf.sublinear_tf = value
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     

       @property
     
    
   

    

     

    
    

     
 def idf_(self):
     
    
   

    

     

    
    

     
 return self._tfidf.idf_
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     
 def fit(self, raw_documents, y=None):
     
    
   

    

     

    
    

     
 """Learn vocabulary and idf from training set.
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Parameters
     
    
   

    

     

    
    

     

       ----------
     
    
   

    

     

    
    

     

       raw_documents : iterable
     
    
   

    

     

    
    

     

       an iterable which yields either str, unicode or file objects
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Returns
     
    
   

    

     

    
    

     

       -------
     
    
   

    

     

    
    

     

       self : TfidfVectorizer
     
    
   

    

     

    
    

     

       """
     
    
   

    

     

    
    

     

       X = super(TfidfVectorizer, self).fit_transform(raw_documents)
     
    
   

    

     

    
    

     

       self._tfidf.fit(X)
     
    
   

    

     

    
    

     
 return self
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     
 def fit_transform(self, raw_documents, y=None):
     
    
   

    

     

    
    

     
 """Learn vocabulary and idf, return term-document matrix.
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       This is equivalent to fit followed by transform, but more efficiently
     
    
   

    

     

    
    

     

       implemented.
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Parameters
     
    
   

    

     

    
    

     

       ----------
     
    
   

    

     

    
    

     

       raw_documents : iterable
     
    
   

    

     

    
    

     

       an iterable which yields either str, unicode or file objects
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Returns
     
    
   

    

     

    
    

     

       -------
     
    
   

    

     

    
    

     

       X : sparse matrix, [n_samples, n_features]
     
    
   

    

     

    
    

     

       Tf-idf-weighted document-term matrix.
     
    
   

    

     

    
    

     

       """
     
    
   

    

     

    
    

     

       X = super(TfidfVectorizer, self).fit_transform(raw_documents)
     
    
   

    

     

    
    

     

       self._tfidf.fit(X)
     
    
   

    

     

    
    

     
 # X is already a transformed view of raw_documents so
     
    
   

    

     

    
    

     
 # we set copy to False
     
    
   

    

     

    
    

     
 return self._tfidf.transform(X, copy=False)
     
    
   

    

     

    
    

     
 
     
    
   

    

     

    
    

     
 def transform(self, raw_documents, copy=True):
     
    
   

    

     

    
    

     
 """Transform documents to document-term matrix.
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Uses the vocabulary and document frequencies (df) learned by fit (or
     
    
   

    

     

    
    

     

       fit_transform).
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Parameters
     
    
   

    

     

    
    

     

       ----------
     
    
   

    

     

    
    

     

       raw_documents : iterable
     
    
   

    

     

    
    

     

       an iterable which yields either str, unicode or file objects
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       copy : boolean, default True
     
    
   

    

     

    
    

     

       Whether to copy X and operate on the copy or perform in-place
     
    
   

    

     

    
    

     

       operations.
     
    
   

    

     

    
    

     

      
     
    
   

    

     

    
    

     

       Returns
     
    
   

    

     

    
    

     

       -------
     
    
   

    

     

    
    

     

       X : sparse matrix, [n_samples, n_features]
     
    
   

    

     

    
    

     

       Tf-idf-weighted document-term matrix.
     
    
   

    

     

    
    

     

       """
     
    
   

    

     

    
    

     

       check_is_fitted(self, '_tfidf', 'The tfidf vector is not fitted')
     
    
   

    

     

    
    

     

       X = super(TfidfVectorizer, self).transform(raw_documents)
     
    
   

    

     

    
    

     
 return self._tfidf.transform(X, copy=False)
     
    
  
 

文章来源: yunyaniu.blog.csdn.net，作者：一个处女座的程序猿，版权归原作者所有，如需转载，请联系作者。

原文链接：yunyaniu.blog.csdn.net/article/details/88086335