OwlCyberSecurity - MANAGER
Edit File: _tokenizer.cpython-38.pyc
U ��.e$+ � @ s� d dl mZmZmZ d dlmZ d dlmZ ddl m Z ddl m Z ddl m Z m Z ddl mZmZmZ dd l mZmZ dd l mZ dd lmZ dd lmZ ee �ZG d d� de�ZdS )� )�absolute_import�division�unicode_literals)�unichr)�deque� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)� tokenTypes� tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Triec sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d d �Zd d � Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )�� HTMLTokenizera This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sF t |f|�| _|| _d| _g | _| j| _d| _d | _t t | �� � d S )NF) r �stream�parserZ escapeFlagZ lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs�� __class__� �C/usr/lib/python3.8/site-packages/pip/_vendor/html5lib/_tokenizer.pyr " s zHTMLTokenizer.__init__c c sP t g �| _| �� rL| jjr6td | jj�d�d�V q| jr | j�� V q6q dS )z� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. � ParseErrorr ��type�dataN)r � tokenQueuer r �errorsr �pop�popleft�r r! r! r" �__iter__1 s zHTMLTokenizer.__iter__c % C s t }d}|rt}d}g }| j�� }||krH|tk rH|�|� | j�� }q"td�|�|�}|tkr�t| }| j �t d dd|id�� �nbd| kr�d ks�n |d kr�d }| j �t d dd|id�� �n d | kr�d k�s�n d| kr�dk�s�n d| k�rdk�s�n d| k�r,dk�s�n |t ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d g#�k�r�| j �t d dd|id�� z t |�}W n> t k �r� |d6 }t d|d? B �t d7|d8@ B � }Y nX |d9k�r| j �t d d:d;�� | j�|� |S )<z�This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. � � � r# z$illegal-codepoint-for-numeric-entity� charAsInt�r% r& Zdatavarsi � i�� i�� � �r � � � � � i� i� � i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i i � i� �;z numeric-entity-without-semicolonr$ )r r r �charr �append�int�joinr r' r � frozenset�chr� ValueError�unget) r ZisHexZallowed�radix� charStack�cr0 r: �vr! r! r" �consumeNumberEntityA s� � � � �� � � � � �� � & � z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j �� g}|d tksB|d tddfksB|d k rV||d krV| j �|d � �n |d dk�r d}|�| j �� � |d dkr�d}|�| j �� � |r�|d tks�|s�|d tkr�| j �|d � | �|�}n4| j �t d d d �� | j �|� � � dd � |� }�nh|d tk �rDt �d � |���s0�qD|�| j �� � �q z$t �d � |d d� ��}t|�}W n tk �r� d }Y nX |d k �r@|d d k�r�| j �t d dd �� |d d k�r|�r|| tk�s�|| tk�s�|| dk�r| j �|� � � dd � |� }n.t| }| j �|� � � |d � ||d � �7 }n4| j �t d dd �� | j �|� � � dd � |� }|�r�| jd d d |7 < n*|tk�r�d}nd}| j �t | |d �� d S )N�&r �<�#F���)�x�XTr# zexpected-numeric-entityr$ r/ r9 z named-entity-without-semicolon�=zexpected-named-entityr& r �SpaceCharacters� Characters)r r: r r rA r; r r rF r' r r) r= � entitiesTrieZhas_keys_with_prefixZlongest_prefix�len�KeyErrorr r r ) r � allowedChar� fromAttribute�outputrC �hexZ entityNameZ entityLengthZ tokenTyper! r! r" � consumeEntity� s~ � � � � � � � � � � zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState". T)rS rT N)rW )r rS r! r! r" �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tkrp|d �t�|d< |d td krp|d rR| j�td dd�� |d rp| j�td d d�� | j�|� | j| _d S ) z�This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r% �name�EndTagr& r# zattributes-in-end-tagr$ � selfClosingz self-closing-flag-on-end-tagN) r r � translater r r' r; r r )r �tokenr! r! r" �emitCurrentToken� s � � z HTMLTokenizer.emitCurrentTokenc C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|dkrd| j�td dd�� | j�td dd�� n`|tkrpdS |t kr�| j�td || j � t d � d�� n&| j � d �}| j�td || d�� d S ) NrG rH � r# �invalid-codepointr$ rO FrN T�rG rH r_ ) r r: �entityDataStater � tagOpenStater' r; r r r � charsUntil�r r& �charsr! r! r" r � s. � � � �zHTMLTokenizer.dataStatec C s | � � | j| _dS �NT)rW r r r+ r! r! r" rb s z HTMLTokenizer.entityDataStatec C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|tkr:dS |dkrp| j�td dd�� | j�td d d�� nT|t kr�| j�td || j � t d � d�� n&| j � d �}| j�td || d�� d S ) NrG rH Fr_ r# r` r$ rO r2 rN Tra ) r r: �characterReferenceInRcdatar �rcdataLessThanSignStater r' r; r r rd re r! r! r" � rcdataState s. � � � �zHTMLTokenizer.rcdataStatec C s | � � | j| _dS rg )rW rj r r+ r! r! r" rh 1 s z(HTMLTokenizer.characterReferenceInRcdatac C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d S � NrH r_ r# r` r$ rO r2 F)rH r_ T) r r: �rawtextLessThanSignStater r' r; r r rd re r! r! r" � rawtextState6 s" � � �zHTMLTokenizer.rawtextStatec C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d S rk ) r r: �scriptDataLessThanSignStater r' r; r r rd re r! r! r" �scriptDataStateH s" � � �z HTMLTokenizer.scriptDataStatec C sr | j �� }|tkrdS |dkrL| j�td dd�� | j�td dd�� n"| j�td || j �d� d�� dS ) NFr_ r# r` r$ rO r2 T)r r: r r' r; r rd �r r& r! r! r" �plaintextStateZ s � � �z HTMLTokenizer.plaintextStatec C s | j �� }|dkr | j| _n�|dkr.| j| _n�|tkrVtd |g ddd�| _| j| _n�|dkr�| j � td dd �� | j � td d d �� | j | _nt|d kr�| j � td d d �� | j � |� | j | _n@| j � td dd �� | j � td dd �� | j � |� | j | _dS )N�!�/ZStartTagF)r% rY r& r[ ZselfClosingAcknowledged�>r# z'expected-tag-name-but-got-right-bracketr$ rO z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerH T)r r: �markupDeclarationOpenStater �closeTagOpenStater r r � tagNameStater' r; r rA �bogusCommentStaterp r! r! r" rc i s@ � � � � zHTMLTokenizer.tagOpenStatec C s� | j �� }|tkr0td |g dd�| _| j| _n�|dkrX| j�td dd�� | j | _nn|t kr�| j�td dd�� | j�td d d�� | j | _n0| j�td d d |id �� | j � |� | j | _dS )NrZ F�r% rY r& r[ rt r# z*expected-closing-tag-but-got-right-bracketr$ z expected-closing-tag-but-got-eofrO �</z!expected-closing-tag-but-got-charr& r1 T) r r: r r r rx r r' r; r r rA ry rp r! r! r" rw � s2 � � � � z HTMLTokenizer.closeTagOpenStatec C s� | j �� }|tkr | j| _n�|dkr.| �� n~|tkrV| j�t d dd�� | j | _nV|dkrh| j | _nD|dkr�| j�t d dd�� | j d d 7 < n| j d |7 < d S ) Nrt r# zeof-in-tag-namer$ rs r_ r` rY r2 T) r r: r �beforeAttributeNameStater r^ r r' r; r r �selfClosingStartTagStater rp r! r! r" rx � s& � �zHTMLTokenizer.tagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS �Nrs r/ rO rH r$ T) r r: �temporaryBuffer�rcdataEndTagOpenStater r' r; r rA rj rp r! r! r" ri � s z%HTMLTokenizer.rcdataLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS �NrO r{ r$ T) r r: r r �rcdataEndTagNameStater r' r; r rA rj rp r! r! r" r� � s z#HTMLTokenizer.rcdataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t kr�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S � NrY rZ Frz rs rt rO r{ r$ T)r �lowerr r r: r r r| r r} r^ r r r' r; rA rj �r Z appropriater& r! r! r" r� � s@ � � � � z#HTMLTokenizer.rcdataEndTagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS r~ ) r r: r �rawtextEndTagOpenStater r' r; r rA rm rp r! r! r" rl � s z&HTMLTokenizer.rawtextLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r r: r r �rawtextEndTagNameStater r' r; r rA rm rp r! r! r" r� � s z$HTMLTokenizer.rawtextEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t kr�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r r r: r r r| r r} r^ r r r' r; rA rm r� r! r! r" r� s@ � � � � z$HTMLTokenizer.rawtextEndTagNameStatec C sx | j �� }|dkr"d| _| j| _nR|dkrJ| j�td dd�� | j| _n*| j�td dd�� | j � |� | j | _dS ) Nrs r/ rr rO z<!r$ rH T) r r: r �scriptDataEndTagOpenStater r' r; r �scriptDataEscapeStartStaterA ro rp r! r! r" rn s z)HTMLTokenizer.scriptDataLessThanSignStatec C sX | j �� }|tkr*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r r: r r �scriptDataEndTagNameStater r' r; r rA ro rp r! r! r" r� , s z'HTMLTokenizer.scriptDataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t kr�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r r r: r r r| r r} r^ r r r' r; rA ro r� r! r! r" r� 7 s@ � � � � z'HTMLTokenizer.scriptDataEndTagNameStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS �N�-rO r$ T) r r: r' r; r � scriptDataEscapeStartDashStater rA ro rp r! r! r" r� S s z(HTMLTokenizer.scriptDataEscapeStartStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS r� ) r r: r' r; r � scriptDataEscapedDashDashStater rA ro rp r! r! r" r� ] s z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkrz| j�td dd�� | j�td dd�� n8|tkr�| j | _n&| j � d �}| j�td || d�� d S ) Nr� rO r$ rH r_ r# r` r2 )rH r� r_ T) r r: r' r; r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r rd re r! r! r" �scriptDataEscapedStateg s( � � �z$HTMLTokenizer.scriptDataEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkr�| j�td dd�� | j�td dd�� | j| _n0|t kr�| j | _n | j�td |d�� | j| _d S ) Nr� rO r$ rH r_ r# r` r2 T) r r: r' r; r r� r r� r� r r rp r! r! r" r� { s& � � z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j �� }|dkr*| j�td dd�� n�|dkr<| j| _n�|dkrd| j�td dd�� | j| _nn|dkr�| j�td dd�� | j�td d d�� | j| _n0|t kr�| j | _n | j�td |d�� | j| _d S ) Nr� rO r$ rH rt r_ r# r` r2 T) r r: r' r; r r� r ro r� r r rp r! r! r" r� � s* � � z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j �� }|dkr"d| _| j| _n\|tkrT| j�td d| d�� || _| j | _n*| j�td dd�� | j � |� | j | _dS r~ ) r r: r � scriptDataEscapedEndTagOpenStater r r' r; r � scriptDataDoubleEscapeStartStaterA r� rp r! r! r" r� � s z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sP | j �� }|tkr"|| _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r r: r r � scriptDataEscapedEndTagNameStater r' r; r rA r� rp r! r! r" r� � s z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tkrT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t kr�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r r r: r r r| r r} r^ r r r' r; rA r� r� r! r! r" r� � s@ � � � � z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C s� | j �� }|ttd�B krR| j�td |d�� | j�� dkrH| j | _ q�| j | _ nB|t kr�| j�td |d�� | j|7 _n| j � |� | j | _ dS �N)rs rt rO r$ ZscriptT)r r: r r> r' r; r r r� � scriptDataDoubleEscapedStater r� r rA rp r! r! r" r� � s z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _nt|dkr�| j�td dd�� | j�td dd�� n>|tkr�| j�td d d�� | j | _n| j�td |d�� d S � Nr� rO r$ rH r_ r# r` r2 �eof-in-script-in-scriptT) r r: r' r; r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r rp r! r! r" r� � s* � � � z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td dd�� | j| _nF|t kr�| j�td d d�� | j | _n | j�td |d�� | j| _d S r� ) r r: r' r; r �$scriptDataDoubleEscapedDashDashStater r� r� r r rp r! r! r" r� s. � � � z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr*| j�td dd�� n�|dkrR| j�td dd�� | j| _n�|dkrz| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td d d�� | j| _nF|t kr�| j�td d d�� | j | _n | j�td |d�� | j| _d S ) Nr� rO r$ rH rt r_ r# r` r2 r� T) r r: r' r; r r� r ro r� r r rp r! r! r" r� s2 � � � z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j �� }|dkr8| j�td dd�� d| _| j| _n| j �|� | j | _dS )Nrs rO r$ r/ T) r r: r' r; r r � scriptDataDoubleEscapeEndStater rA r� rp r! r! r" r� 0 s z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j �� }|ttd�B krR| j�td |d�� | j�� dkrH| j | _ q�| j | _ nB|t kr�| j�td |d�� | j|7 _n| j � |� | j | _ dS r� )r r: r r> r' r; r r r� r� r r� r rA rp r! r! r" r� ; s z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j �� }|tkr$| j �td� �n|tkrJ| jd �|dg� | j| _n�|dkr\| � � n�|dkrn| j | _n�|dkr�| j �t d dd �� | jd �|dg� | j| _n�|d kr�| j �t d d d �� | jd �d dg� | j| _nF|t k�r| j �t d d d �� | j| _n | jd �|dg� | j| _dS )NTr& r/ rt rs )�'�"rM rH r# �#invalid-character-in-attribute-namer$ r_ r` r2 z#expected-attribute-name-but-got-eof)r r: r rd r r r; �attributeNameStater r^ r} r' r r r rp r! r! r" r| K s< � � � z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr&| j| _�n.|tkr\| jd d d || j �td� 7 < d}n�|dkrjd}n�|tkr|| j| _n�|dkr�| j | _n�|d kr�| j � t d d d �� | jd d d d 7 < d}n�|dk�r | j � t d dd �� | jd d d |7 < d}nH|t k�r6| j � t d dd �� | j| _n | jd d d |7 < d}|�r�| jd d d �t�| jd d d< | jd d d� D ]>\}}| jd d d |k�r�| j � t d dd �� �qҐq�|�r�| �� dS )NTFrM r& rJ r rt rs r_ r# r` r$ r2 �r� r� rH r� zeof-in-attribute-namezduplicate-attribute)r r: �beforeAttributeValueStater r r rd r �afterAttributeNameStater} r' r; r r r r\ r r^ )r r& ZleavingThisStateZ emitTokenrY �_r! r! r" r� i s^ � � � � � � z HTMLTokenizer.attributeNameStatec C sD | j �� }|tkr$| j �td� �n |dkr8| j| _�n|dkrJ| �� n�|tkrp| jd � |dg� | j | _n�|dkr�| j | _n�|dkr�| j � t d d d �� | jd � d dg� | j | _n�|d kr�| j � t d d d �� | jd � |dg� | j | _nF|tk�r$| j � t d dd �� | j| _n | jd � |dg� | j | _dS )NTrM rt r& r/ rs r_ r# r` r$ r2 r� z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r r: r rd r� r r^ r r r; r� r} r' r r r rp r! r! r" r� � s@ � � � z%HTMLTokenizer.afterAttributeNameStatec C sh | j �� }|tkr$| j �td� �n@|dkr8| j| _�n,|dkrX| j| _| j �|� �n |dkrj| j| _n�|dkr�| j � t d dd�� | � � n�|d kr�| j � t d d d�� | j d d d d7 < | j| _n�|dk�r| j � t d dd�� | j d d d |7 < | j| _nL|tk�rB| j � t d dd�� | j| _n"| j d d d |7 < | j| _dS )NTr� rG r� rt r# z.expected-attribute-value-but-got-right-bracketr$ r_ r` r& rJ r r2 )rM rH �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r r: r rd � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterA � attributeValueSingleQuotedStater' r; r r^ r r r rp r! r! r" r� � sF � � � � z'HTMLTokenizer.beforeAttributeValueStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d 7 < nN|t kr�| j�td d d�� | j | _n&| jd d d || j � d � 7 < d S )Nr� rG r_ r# r` r$ r& rJ r r2 z#eof-in-attribute-value-double-quote)r� rG r_ T� r r: �afterAttributeValueStater rX r' r; r r r r rd rp r! r! r" r� � s& � � �z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d 7 < nN|t kr�| j�td d d�� | j | _n&| jd d d || j � d � 7 < d S )Nr� rG r_ r# r` r$ r& rJ r r2 z#eof-in-attribute-value-single-quote)r� rG r_ Tr� rp r! r! r" r� � s& � � �z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr0| �d� n�|dkrB| �� n�|dkr|| j�t d dd�� | j d d d |7 < n�|d kr�| j�t d d d�� | j d d d d 7 < nV|t kr�| j�t d d d�� | j | _n.| j d d d || j � td�tB � 7 < dS )NrG rt )r� r� rM rH r� r# z0unexpected-character-in-unquoted-attribute-valuer$ r& rJ r r_ r` r2 z eof-in-attribute-value-no-quotes)rG rt r� r� rM rH r� r_ T)r r: r r| r rX r^ r' r; r r r r rd r> rp r! r! r" r� s4 � � � � z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tkr | j| _n�|dkr.| �� np|dkr@| j| _n^|tkrt| j� t d dd�� | j � |� | j | _n*| j� t d dd�� | j � |� | j| _dS )Nrt rs r# z$unexpected-EOF-after-attribute-valuer$ z*unexpected-character-after-attribute-valueT) r r: r r| r r^ r} r r' r; r rA r rp r! r! r" r� s& � � z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkr&d| jd< | �� n^|tkrZ| j�td dd�� | j �|� | j | _ n*| j�td dd�� | j �|� | j | _ dS )Nrt Tr[ r# z#unexpected-EOF-after-solidus-in-tagr$ z)unexpected-character-after-solidus-in-tag) r r: r r^ r r' r; r rA r r r| rp r! r! r" r} 4 s � � z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nrt r_ r2 �Commentr$ T) r rd �replacer' r; r r: r r rp r! r! r" ry F s � z HTMLTokenizer.bogusCommentStatec C s� | j �� g}|d dkrR|�| j �� � |d dkrPtd dd�| _| j| _dS n�|d dkr�d}dD ](}|�| j �� � |d |krfd } q�qf|r�td dd d dd �| _| j| _dS n�|d d k�rD| jd k �rD| jj j �rD| jj j d j | jj j k�rDd}d D ].}|�| j �� � |d |k�rd } �q2�q|�rD| j | _dS | j�td dd�� |�rt| j �|�� � �qZ| j| _dS )NrJ r� r� r/ r$ T)�d�D))�o�O�rD �C��t�T��y�Y��p�P��e�EFZDoctype)r% rY �publicId�systemId�correct�[)r� r� �Ar� r� r� r# zexpected-dashes-or-doctype)r r: r; r r �commentStartStater � doctypeStater ZtreeZ openElements� namespaceZdefaultNamespace�cdataSectionStater' rA r) ry )r rC �matched�expectedr! r! r" rv U s\ ���� �z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nP|t kr�| j�td d d�� | j�| j� | j| _n| jd |7 < | j | _d S ) Nr� r_ r# r` r$ r& r2 rt �incorrect-comment�eof-in-commentT) r r: �commentStartDashStater r' r; r r r r � commentStaterp r! r! r" r� � s. � � � z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nT|t kr�| j�td d d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nr� r_ r# r` r$ r& � -�rt r� r� T) r r: �commentEndStater r' r; r r r r r� rp r! r! r" r� � s. � � � z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < nT|tkr�| j�td dd�� | j�| j� | j | _n | jd || j � d � 7 < d S ) Nr� r_ r# r` r$ r& r2 r� )r� r_ T) r r: �commentEndDashStater r' r; r r r r rd rp r! r! r" r� � s$ � � �zHTMLTokenizer.commentStatec C s� | j �� }|dkr | j| _n�|dkrV| j�td dd�� | jd d7 < | j| _nT|t kr�| j�td dd�� | j�| j� | j | _n | jd d| 7 < | j| _d S ) Nr� r_ r# r` r$ r& r� zeof-in-comment-end-dashT) r r: r� r r' r; r r r� r r rp r! r! r" r� � s$ � � z!HTMLTokenizer.commentEndDashStatec C s, | j �� }|dkr*| j�| j� | j| _n�|dkrd| j�td dd�� | jd d7 < | j| _n�|dkr�| j�td d d�� | j | _n�|d kr�| j�td d d�� | jd |7 < nj|t kr�| j�td d d�� | j�| j� | j| _n4| j�td d d�� | jd d| 7 < | j| _dS )Nrt r_ r# r` r$ r& u --�rr z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r: r' r; r r r r r� �commentEndBangStater rp r! r! r" r� � s@ � � � � �z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr*| j�| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| j�td dd�� | jd d 7 < | j | _nT|t kr�| j�td d d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nrt r� r& z--!r_ r# r` r$ u --!�z eof-in-comment-end-bang-stateT) r r: r' r; r r r r� r r� r rp r! r! r" r� � s, � � z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tkr | j| _nj|tkr\| j�td dd�� d| j d< | j�| j � | j | _n*| j�td dd�� | j � |� | j| _dS )Nr# �!expected-doctype-name-but-got-eofr$ Fr� zneed-space-after-doctypeT) r r: r �beforeDoctypeNameStater r r' r; r r r rA rp r! r! r" r� s � � zHTMLTokenizer.doctypeStatec C s� | j �� }|tkrn�|dkrT| j�td dd�� d| jd< | j�| j� | j| _n�|dkr�| j�td dd�� d | jd < | j | _nR|t kr�| j�td d d�� d| jd< | j�| j� | j| _n|| jd < | j | _d S ) Nrt r# z+expected-doctype-name-but-got-right-bracketr$ Fr� r_ r` r2 rY r� T) r r: r r' r; r r r r �doctypeNameStater rp r! r! r" r� s4 � � � z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tkr2| jd �t�| jd< | j| _n�|dkrh| jd �t�| jd< | j� | j� | j | _n�|dkr�| j� t d dd�� | jd d7 < | j | _nh|t kr�| j� t d dd�� d | jd < | jd �t�| jd< | j� | j� | j | _n| jd |7 < d S ) NrY rt r_ r# r` r$ r2 zeof-in-doctype-nameFr� T)r r: r r r\ r �afterDoctypeNameStater r' r; r r r� r rp r! r! r" r� 6 s0 � � z HTMLTokenizer.doctypeNameStatec C sH | j �� }|tkr�n.|dkr8| j�| j� | j| _�n |tkr�d| jd< | j � |� | j�t d dd�� | j�| j� | j| _n�|dkr�d}d D ] }| j �� }||kr�d} q�q�|r�| j | _dS nD|d k�r d}d D ] }| j �� }||kr�d} q�q�|�r | j | _dS | j � |� | j�t d d d |id�� d| jd< | j | _dS )Nrt Fr� r# �eof-in-doctyper$ r� T))�u�U)�b�B)�l�L)�i�Ir� ��s�S)r� r� r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper& r1 )r r: r r' r; r r r r rA r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r r& r� r� r! r! r" r� O sT � � z#HTMLTokenizer.afterDoctypeNameStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j d< | j�| j � | j | _n| j �|� | j| _dS � N)r� r� r# �unexpected-char-in-doctyper$ r� Fr� T) r r: r �"beforeDoctypePublicIdentifierStater r' r; r rA r r r rp r! r! r" r� � s& � � z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd < | j�| j� | j | _nh|t kr�| j�t d d d�� d | jd < | j�| j� | j | _n(| j�t d d d�� d | jd < | j | _d S )Nr� r/ r� r� rt r# �unexpected-end-of-doctyper$ Fr� r� r� T) r r: r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater' r; r r r r� rp r! r! r" r� � s: � � � z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t kr�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� T� r r: �!afterDoctypePublicIdentifierStater r' r; r r r r rp r! r! r" r� � s0 � � � z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t kr�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� Tr� rp r! r! r" r� � s0 � � � z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr<| j�| j� | j| _n�|dkrn| j�t d dd�� d| jd< | j | _n�|dkr�| j�t d dd�� d| jd< | j | _nh|t kr�| j�t d d d�� d | jd < | j�| j� | j| _n(| j�t d dd�� d | jd < | j | _d S ) Nrt r� r# r� r$ r/ r� r� r� Fr� T)r r: r �-betweenDoctypePublicAndSystemIdentifiersStater r' r; r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� rp r! r! r" r� � s> � � � � z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tkrn�|dkr4| j�| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t kr�| j�t d dd�� d | jd < | j�| j� | j| _n(| j�t d d d�� d | jd < | j | _d S ) Nrt r� r/ r� r� r# r� r$ Fr� r� T) r r: r r' r; r r r r� r� r r r� rp r! r! r" r� s2 � � z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j d< | j�| j � | j | _n| j �|� | j| _dS r� ) r r: r �"beforeDoctypeSystemIdentifierStater r' r; r rA r r r rp r! r! r" r� s& � � z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd < | j�| j� | j | _nh|t kr�| j�t d d d�� d | jd < | j�| j� | j | _n(| j�t d dd�� d | jd < | j | _d S ) Nr� r/ r� r� rt r# r� r$ Fr� r� T) r r: r r r� r r� r' r; r r r r� rp r! r! r" r� / s: � � � z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t kr�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� T� r r: �!afterDoctypeSystemIdentifierStater r' r; r r r r rp r! r! r" r� L s0 � � � z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t kr�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� Tr� rp r! r! r" r� d s0 � � � z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tkrn~|dkr4| j�| j� | j| _n^|tkrt| j�t d dd�� d| jd< | j�| j� | j| _n | j�t d dd�� | j | _dS ) Nrt r# r� r$ Fr� r� T) r r: r r' r; r r r r r r� rp r! r! r" r� | s$ � �z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j �� }|dkr*| j�| j� | j| _n,|tkrV| j �|� | j�| j� | j| _n dS )Nrt T) r r: r' r; r r r r rA rp r! r! r" r� � s z HTMLTokenizer.bogusDoctypeStatec C s� g }|� | j�d�� |� | j�d�� | j�� }|tkr>q�q|dksJt�|d dd � dkrv|d d d� |d< q�q|� |� qd�|�}|�d�}|dkr�t|�D ]}| j � t d d d �� q�|� dd �}|r�| j � t d |d �� | j | _ dS )N�]rt rJ ���z]]r/ r_ r r# r` r$ r2 rO T)r; r rd r: r �AssertionErrorr= �count�ranger' r r� r r )r r& r: Z nullCountr� r! r! r" r� � s2 � �z HTMLTokenizer.cdataSectionState)N)NF)N�__name__� __module__� __qualname__�__doc__r r, rF rW rX r^ r rb rj rh rm ro rq rc rw rx ri r� r� rl r� r� rn r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r| r� r� r� r� r� r� r� r} ry rv r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� � __classcell__r! r! r r" r s� H P # 6 "- 3 r N)Z __future__r r r Zpip._vendor.sixr r? � collectionsr Z constantsr r r r r r r r r r Z _inputstreamr Z_trier rP �objectr r! r! r! r" �<module> s