OwlCyberSecurity - MANAGER
Edit File: universaldetector.cpython-38.pyc
U ʗRe�3 � @ s� d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl m Z ddl m Z ddlmZ dd lmZ dd lmZ G d d � d �ZdS ) a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco � N� )�CharSetGroupProber)� InputState�LanguageFilter� ProbingState)�EscCharSetProber)� Latin1Prober)�MBCSGroupProber)�SBCSGroupProber)� UTF1632Proberc @ s� e Zd ZdZdZe�d�Ze�d�Ze�d�Z dddd d d d d d�Z e j fdd�Z edd� �Zedd� �Zedd� �Zdd� Zdd� Zdd � Zd S ) �UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g�������?s [�-�]s (|~{)s [�-�]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9z iso-8859-13c C sT d | _ d | _g | _d | _d | _d | _d | _d | _|| _t � t �| _ d | _ | �� d S �N)�_esc_charset_prober�_utf1632_prober�_charset_probers�result�done� _got_data� _input_state� _last_char� lang_filter�logging� getLogger�__name__�logger�_has_win_bytes�reset)�selfr � r ��/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/universaldetector.py�__init__T s zUniversalDetector.__init__c C s | j S r )r �r r r r � input_stateb s z UniversalDetector.input_statec C s | j S r )r r! r r r � has_win_bytesf s z UniversalDetector.has_win_bytesc C s | j S r )r r! r r r �charset_probersj s z!UniversalDetector.charset_probersc C sf dddd�| _ d| _d| _d| _tj| _d| _| jr>| j� � | j rN| j � � | j D ] }|� � qTdS )z� Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. N� ��encoding� confidence�languageF� ) r r r r r � PURE_ASCIIr r r r r r )r �proberr r r r n s zUniversalDetector.resetc C s� | j r dS |sdS t|t�s$t|�}| js�|�tj�rFdddd�| _nv|�tjtj f�rhdddd�| _nT|�d�r�dddd�| _n:|�d �r�d ddd�| _n |�tj tj f�r�d ddd�| _d | _| jd dk r�d | _ dS | j t jk�r(| j�|�r�t j| _ n*| j t jk�r(| j�| j| ��r(t j| _ |dd� | _| j�sFt� | _| jjtjk�r�| j�|�tjk�r�| jj | j� � dd�| _d | _ dS | j t jk�r�| j �s�t | j �| _ | j �|�tjk�r�| j j | j � � | j j!d�| _d | _ n�| j t jk�r�| j"�s4t#| j �g| _"| j t$j%@ �r&| j"�&t'� � | j"�&t(� � | j"D ]:}|�|�tjk�r:|j |� � |j!d�| _d | _ �qv�q:| j)�|��r�d | _*dS )a� Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIG� �?� r& zUTF-32s �� zX-ISO-10646-UCS-4-3412s ��zX-ISO-10646-UCS-4-2143zUTF-16Tr' ���)+r � isinstance� bytearrayr � startswith�codecs�BOM_UTF8r � BOM_UTF32_LE� BOM_UTF32_BE�BOM_LE�BOM_BEr r r+ �HIGH_BYTE_DETECTOR�search� HIGH_BYTE� ESC_DETECTORr � ESC_ASCIIr r �stater � DETECTING�feed�FOUND_IT� charset_name�get_confidencer r r r) r r r �NON_CJK�appendr r �WIN_BYTE_DETECTORr )r �byte_strr, r r r r@ � s� � � � ��� � � zUniversalDetector.feedc C st | j r | jS d| _ | js&| j�d� n�| jtjkrBdddd�| _n�| jtjkr�d}d}d}| j D ]"}|sjq`|� � }||kr`|}|}q`|r�|| j kr�|j }|j � � }|� � }|�d �r�| jr�| j�||�}|||jd�| _| j�� tjk�rn| jd dk�rn| j�d � | j D ]`}|�s�q t|t��rP|jD ] }| j�d |j |j|� � � �q,n| j�d |j |j|� � � �q | jS ) z� Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!�asciir- r. r& Nr% ziso-8859r' z no probers hit minimum thresholdz%s %s confidence = %s)r r r r �debugr r r+ r; r rC �MINIMUM_THRESHOLDrB �lowerr2 r � ISO_WIN_MAP�getr) �getEffectiveLevelr �DEBUGr0 r �probers) r �prober_confidence�max_prober_confidence� max_proberr, rB �lower_charset_namer( � group_proberr r r �close sj �� � �zUniversalDetector.closeN)r � __module__� __qualname__�__doc__rJ �re�compiler9 r< rF rL r �ALLr �propertyr" r# r$ r r@ rV r r r r r 4 s2 � r )rY r3 r rZ �charsetgroupproberr �enumsr r r � escproberr � latin1proberr �mbcsgroupproberr �sbcsgroupproberr � utf1632proberr r r r r r �<module> s