U >iّ@sdZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl m Zddl mZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZGdd d e Z!Gd d d e!Z"Gd d d e!Z#Gddde!ej$Z$ddZ%ddZ&ddZ'ddZ(ddZ)dS)zVirtual bases classes for uploading media via Google APIs. Supported here are: * simple (media) uploads * multipart uploads that contain both metadata and a small file as payload * resumable uploads (with metadata as well) N)_async_resumable_media)_helpers)_upload)common)_CONTENT_TYPE_HEADER_CONTENT_RANGE_TEMPLATE_RANGE_UNKNOWN_TEMPLATE_EMPTY_RANGE_TEMPLATE_BOUNDARY_FORMAT_MULTIPART_SEP_CRLF_MULTIPART_BEGIN_RELATED_HEADER_BYTES_RANGE_RE_STREAM_ERROR_TEMPLATE_POST_PUT!_UPLOAD_CHECKSUM_MISMATCH_MESSAGE0_UPLOAD_METADATA_NO_APPROPRIATE_CHECKSUM_MESSAGEc@sReZdZdZdddZeddZddZed d Z ed d Z ed dZ dS) UploadBaseaBase class for upload helpers. Defines core shared behavior across different upload types. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. Attributes: upload_url (str): The URL where the content will be uploaded. NcCs,||_|dkri}||_d|_t|_dS)NF) upload_url_headers _finishedrZ RetryStrategyZ_retry_strategy)selfrheadersrI/tmp/pip-unpacked-wheel-stw6o2t8/google/_async_resumable_media/_upload.py__init__Fs zUploadBase.__init__cCs|jS)z2bool: Flag indicating if the upload has completed.)rrrrrfinishedNszUploadBase.finishedcCs d|_t|tjjf|jdS)aProcess the response from an HTTP request. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is not 200. .. _sans-I/O: https://sans-io.readthedocs.io/ TN)rrrequire_status_codehttpclientOK_get_status_coderresponserrr_process_responseSszUploadBase._process_responsecCs tddS)zAccess the status code from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. This implementation is virtual.NNotImplementedErrorr&rrrr$hs zUploadBase._get_status_codecCs tddS)zAccess the headers from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. r(Nr)r+rrr _get_headersts zUploadBase._get_headerscCs tddS)zAccess the response body from an HTTP response. Args: response (object): The HTTP response object. Raises: NotImplementedError: Always, since virtual. r(Nr)r+rrr _get_bodys zUploadBase._get_body)N) __name__ __module__ __qualname____doc__rpropertyrr' staticmethodr$r,r-rrrrr8s    rc@s"eZdZdZddZdddZdS) SimpleUploadaUpload a resource to a Google API. A **simple** media upload sends no metadata and completes the upload in a single request. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. Attributes: upload_url (str): The URL where the content will be uploaded. cCs@|jrtdt|ts&tdt|||jt<t|j ||jfS)aPrepare the contents of an HTTP request. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. .. note: This method will be used only once, so ``headers`` will be mutated by having a new key added to it. Args: data (bytes): The resource content to be uploaded. content_type (str): The content type for the request. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already finished. TypeError: If ``data`` isn't bytes. .. _sans-I/O: https://sans-io.readthedocs.io/ An upload can only be used once.`data` must be bytes, received) r ValueError isinstancebytes TypeErrortyperrrr)rdata content_typerrr_prepare_requests   zSimpleUpload._prepare_requestNcCs tddS)a Transmit the resource to be uploaded. Args: transport (object): An object which can make authenticated requests. data (bytes): The resource content to be uploaded. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r(Nr))r transportr<r=timeoutrrrtransmitszSimpleUpload.transmit)N)r.r/r0r1r>rArrrrr4s&r4cs4eZdZdZd fdd ZddZd ddZZS) MultipartUploada3Upload a resource with metadata to a Google API. A **multipart** upload sends both metadata and the resource in a single (multipart) request. Args: upload_url (str): The URL where the content will be uploaded. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the request, e.g. headers for encrypted data. checksum Optional([str]): The type of checksum to compute to verify the integrity of the object. The request metadata will be amended to include the computed value. Using this option will override a manually-set checksum value. Supported values are "md5", "crc32c" and None. The default is None. Attributes: upload_url (str): The URL where the content will be uploaded. Ncstt|j||d||_dS)Nr)superrBr_checksum_type)rrrchecksum __class__rrrszMultipartUpload.__init__c Cs|jrtdt|ts&tdt|t|j}|dk rf| |t | }t |j}|||<t |||\}}t|d} | |jt<t|j||jfS)ajPrepare the contents of an HTTP request. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. .. note: This method will be used only once, so ``headers`` will be mutated by having a new key added to it. Args: data (bytes): The resource content to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already finished. TypeError: If ``data`` isn't bytes. .. _sans-I/O: https://sans-io.readthedocs.io/ r5r6N")rr7r8r9r:r; sync_helpersZ_get_checksum_objectrEupdateprepare_checksum_digestdigest_get_metadata_keyconstruct_multipart_requestrrrrr) rr<metadatar=Zchecksum_objectZactual_checksum metadata_keycontentmultipart_boundaryZmultipart_content_typerrrr>s(!      z MultipartUpload._prepare_requestcCs tddS)aqTransmit the resource to be uploaded. Args: transport (object): An object which can make authenticated requests. data (bytes): The resource content to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r(Nr))rr?r<rPr=r@rrrrA)szMultipartUpload.transmit)NN)N)r.r/r0r1rr>rA __classcell__rrrGrrBs:rBcseZdZdZd&fdd ZeddZeddZed d Zed d Z ed dZ d'ddZ ddZ d(ddZ ddZddZddZddZd)ddZd d!Zd"d#Zd$d%ZZS)*ResumableUploadaInitiate and fulfill a resumable upload to a Google API. A **resumable** upload sends an initial request with the resource metadata and then gets assigned an upload ID / upload URL to send bytes to. Using the upload URL, the upload is then done in chunks (determined by the user) until all bytes have been uploaded. Args: upload_url (str): The URL where the resumable upload will be initiated. chunk_size (int): The size of each chunk used to upload the resource. headers (Optional[Mapping[str, str]]): Extra headers that should be sent with the :meth:`initiate` request, e.g. headers for encrypted data. These **will not** be sent with :meth:`transmit_next_chunk` or :meth:`recover` requests. checksum Optional([str]): The type of checksum to compute to verify the integrity of the object. After the upload is complete, the server-computed checksum of the resulting object will be read and google.resumable_media.common.DataCorruption will be raised on a mismatch. The corrupted file will not be deleted from the remote host automatically. Supported values are "md5", "crc32c" and None. The default is None. Attributes: upload_url (str): The URL where the content will be uploaded. Raises: ValueError: If ``chunk_size`` is not a multiple of :data:`.UPLOAD_CHUNK_SIZE`. Ncsvtt|j||d|tjdkr6tdtjd||_d|_d|_ d|_ d|_ ||_ d|_ d|_d|_d|_dS)NrCrz{} KB must divide chunk sizeiF)rDrUrrZUPLOAD_CHUNK_SIZEr7format _chunk_size_stream _content_type_bytes_uploadedZ_bytes_checksummedrE_checksum_object _total_bytes_resumable_url_invalid)rr chunk_sizerFrrGrrr_s"zResumableUpload.__init__cCs|jS)zbool: Indicates if the upload is in an invalid state. This will occur if a call to :meth:`transmit_next_chunk` fails. To recover from such a failure, call :meth:`recover`. r^rrrrinvalidrszResumableUpload.invalidcCs|jS)z8int: The size of each chunk used to upload the resource.)rWrrrrr_{szResumableUpload.chunk_sizecCs|jS)z;Optional[str]: The URL of the in-progress resumable upload.)r]rrrr resumable_urlszResumableUpload.resumable_urlcCs|jS)z-int: Number of bytes that have been uploaded.)rZrrrrbytes_uploadedszResumableUpload.bytes_uploadedcCs|jS)aOptional[int]: The total number of bytes to be uploaded. If this upload is initiated (via :meth:`initiate`) with ``stream_final=True``, this value will be populated based on the size of the ``stream`` being uploaded. (By default ``stream_final=True``.) If this upload is initiated with ``stream_final=False``, :attr:`total_bytes` will be :data:`None` since it cannot be determined from the stream. )r\rrrr total_bytess zResumableUpload.total_bytesTc Cs|jdk rtd|dkr&td||_||_tdd|i}|dk rN||_n|r\t||_|jdk rzd|j}||d<| |j t | d }t|j||fS) aPrepare the contents of HTTP request to initiate upload. This is everything that must be done before a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: stream (IO[bytes]): The stream (i.e. file-like object) that will be uploaded. The stream **must** be at the beginning (i.e. ``stream.tell() == 0``). metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. total_bytes (Optional[int]): The total number of bytes to be uploaded. If specified, the upload size **will not** be determined from the stream (even if ``stream_final=True``). stream_final (Optional[bool]): Indicates if the ``stream`` is "final" (i.e. no more bytes will be added to it). In this case we determine the upload size from the size of the stream. If ``total_bytes`` is passed, this argument will be ignored. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always POST) * the URL for the request * the body of the request * headers for the request Raises: ValueError: If the current upload has already been initiated. ValueError: If ``stream`` is not at the beginning. .. _sans-I/O: https://sans-io.readthedocs.io/ Nz'This upload has already been initiated.rzStream must be at beginning.zapplication/json; charset=UTF-8zx-upload-content-typez{:d}zx-upload-content-lengthutf-8)rbr7tellrXrYrr\get_total_bytesrVrKrjsondumpsencoderr) rstreamrPr=rd stream_finalrcontent_lengthpayloadrrr_prepare_initiate_requests*'      z)ResumableUpload._prepare_initiate_requestcCs2tj|tjjf|j|jdt|d|j|_ dS)aVProcess the response from an HTTP request that initiated upload. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. This method takes the URL from the ``Location`` header and stores it for future use. Within that URL, we assume the ``upload_id`` query parameter has been included, but we do not check. Args: response (object): The HTTP response object (need headers). .. _sans-I/O: https://sans-io.readthedocs.io/ callbacklocationN) rr r!r"r#r$ _make_invalidheader_requiredr,r]r%rrr_process_initiate_responsesz*ResumableUpload._process_initiate_responsecCs tddS)ahInitiate a resumable upload. By default, this method assumes your ``stream`` is in a "final" state ready to transmit. However, ``stream_final=False`` can be used to indicate that the size of the resource is not known. This can happen if bytes are being dynamically fed into ``stream``, e.g. if the stream is attached to application logs. If ``stream_final=False`` is used, :attr:`chunk_size` bytes will be read from the stream every time :meth:`transmit_next_chunk` is called. If one of those reads produces strictly fewer bites than the chunk size, the upload will be concluded. Args: transport (object): An object which can make authenticated requests. stream (IO[bytes]): The stream (i.e. file-like object) that will be uploaded. The stream **must** be at the beginning (i.e. ``stream.tell() == 0``). metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. total_bytes (Optional[int]): The total number of bytes to be uploaded. If specified, the upload size **will not** be determined from the stream (even if ``stream_final=True``). stream_final (Optional[bool]): Indicates if the ``stream`` is "final" (i.e. no more bytes will be added to it). In this case we determine the upload size from the size of the stream. If ``total_bytes`` is passed, this argument will be ignored. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r(Nr))rr?rkrPr=rdrlr@rrrinitiates1zResumableUpload.initiatecCs|jrtd|jrtd|jdkr.tdt|j|j|j\}}}||jkrft ||j}t|| ||t |j tj|i}t|j||fS)aPrepare the contents of HTTP request to upload a chunk. This is everything that must be done before a request that doesn't require network I/O. This is based on the `sans-I/O`_ philosophy. For the time being, this **does require** some form of I/O to read a chunk from ``stream`` (via :func:`get_next_chunk`). However, this will (almost) certainly not be network I/O. Returns: Tuple[str, str, bytes, Mapping[str, str]]: The quadruple * HTTP verb for the request (always PUT) * the URL for the request * the body of the request * headers for the request The headers **do not** incorporate the ``_headers`` on the current instance. Raises: ValueError: If the current upload has finished. ValueError: If the current upload is in an invalid state. ValueError: If the current upload has not been initiated. ValueError: If the location in the stream (i.e. ``stream.tell()``) does not agree with ``bytes_uploaded``. .. _sans-I/O: https://sans-io.readthedocs.io/ zUpload has finished.z;Upload is in an invalid state. To recover call `recover()`.Nz_This upload has not been initiated. Please call initiate() before beginning to transmit chunks.)rr7rarbget_next_chunkrXrWr\rcrrVZ_update_checksumrrYrCONTENT_RANGE_HEADERr)r start_bytern content_rangemsgrrrrr>%s2    z ResumableUpload._prepare_requestcCs d|_dS)zSimple setter for ``invalid``. This is intended to be passed along as a callback to helpers that raise an exception so they can mark this instance as invalid before raising. TNr`rrrrrs^szResumableUpload._make_invalidcstj|tjjtjjf|j|jd}|tjjkrR|j||_d|_ | |IdHnVtj |tj |j |jd}t|}|dkr|t|d|dt|dd|_dS)aProcess the response from an HTTP request. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. bytes_sent (int): The number of bytes sent in the request that ``response`` was returned for. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is 308 and the ``range`` header is not of the form ``bytes 0-{end}``. ~google.resumable_media.common.InvalidResponse: If the status code is not 200 or 308. .. _sans-I/O: https://sans-io.readthedocs.io/ rpTNUnexpected "range" header*Expected to be of the form "bytes=0-{end}"end_byte)rr r!r"r#PERMANENT_REDIRECTr$rsrZr_validate_checksumrt RANGE_HEADERr,rmatchrInvalidResponseintgroup)rr&Z bytes_sent status_code bytes_rangerrrr_process_resumable_responsegs4  z+ResumableUpload._process_resumable_responsecs|jdkrdSt|j}|IdH}||}|dkrTt|t|| |t |j }||krt |t|j||dS)aQCheck the computed checksum, if any, against the response headers. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.DataCorruption: If the checksum computed locally and the checksum reported by the remote host do not match. N)rErJrNrhgetrrrrVr,rLr[rMZDataCorruptionrupper)rr&rQrPZremote_checksumZlocal_checksumrrrrs.   z"ResumableUpload._validate_checksumcCs tddS)aTransmit the next chunk of the resource to be uploaded. If the current upload was initiated with ``stream_final=False``, this method will dynamically determine if the upload has completed. The upload will be considered complete if the stream produces fewer than :attr:`chunk_size` bytes when a chunk is read from it. Args: transport (object): An object which can make authenticated requests. timeout (Optional[Union[float, aiohttp.ClientTimeout]]): The number of seconds to wait for the server response. Depending on the retry strategy, a request may be repeated several times using the same timeout each time. Can also be passed as an `aiohttp.ClientTimeout` object. Raises: NotImplementedError: Always, since virtual. r(Nr))rr?r@rrrtransmit_next_chunksz#ResumableUpload.transmit_next_chunkcCs&|jstdtjdi}t|jd|fS)aPrepare the contents of HTTP request to recover from failure. This is everything that must be done before a request that doesn't require network I/O. This is based on the `sans-I/O`_ philosophy. We assume that the :attr:`resumable_url` is set (i.e. the only way the upload can end up :attr:`invalid` is if it has been initiated. Returns: Tuple[str, str, NoneType, Mapping[str, str]]: The quadruple * HTTP verb for the request (always PUT) * the URL for the request * the body of the request (always :data:`None`) * headers for the request The headers **do not** incorporate the ``_headers`` on the current instance. Raises: ValueError: If the current upload is not in an invalid state. .. _sans-I/O: https://sans-io.readthedocs.io/ z3Upload is not in invalid state, no need to recover.z bytes */*N)rar7rrxrrb)rrrrr_prepare_recover_requests z(ResumableUpload._prepare_recover_requestcCst|tjjf|j||}tj|krl|tj}t |}|dkrVt |d|dt | dd|_nd|_|j|jd|_dS)aProcess the response from an HTTP request to recover from failure. This is everything that must be done after a request that doesn't require network I/O (or other I/O). This is based on the `sans-I/O`_ philosophy. Args: response (object): The HTTP response object. Raises: ~google.resumable_media.common.InvalidResponse: If the status code is not 308. ~google.resumable_media.common.InvalidResponse: If the status code is 308 and the ``range`` header is not of the form ``bytes 0-{end}``. .. _sans-I/O: https://sans-io.readthedocs.io/ Nr|r}r~rrF)rr r!r"rr$r,rrrrrrrrZrXseekr^)rr&rrrrrr_process_recover_responses(    z)ResumableUpload._process_recover_responsecCs tddS)a!Recover from a failure. This method should be used when a :class:`ResumableUpload` is in an :attr:`~ResumableUpload.invalid` state due to a request failure. This will verify the progress with the server and make sure the current upload is in a valid state before :meth:`transmit_next_chunk` can be used again. Args: transport (object): An object which can make authenticated requests. Raises: NotImplementedError: Always, since virtual. r(Nr))rr?rrrrecover!szResumableUpload.recover)NN)NT)NTN)N)r.r/r0r1rr2rar_rbrcrdrorurvr>rsrrrrrrrTrrrGrrU@s8      @  39 ; +rUcCs ttj}t|}|dS)zGet a random boundary for a multipart request. Returns: bytes: The boundary used to separate parts of a multipart request. re)random randrangesysmaxsizer rVrj)Z random_intboundaryrrr get_boundary5s  rcCsht}t|d}|d}t|}|t|t|td|tt|t|t}||fS)aConstruct a multipart request body. Args: data (bytes): The resource content (UTF-8 encoded as bytes) to be uploaded. metadata (Mapping[str, str]): The resource metadata, such as an ACL list. content_type (str): The content type of the resource, e.g. a JPEG image has content type ``image/jpeg``. Returns: Tuple[bytes, bytes]: The multipart request body and the boundary used between each part. rescontent-type: )rrhrirjr r r )r<rPr=rSZ json_bytesZ boundary_seprRrrrrOBsB      rOcCs,|}|dtj|}|||S)zDetermine the total number of bytes in a stream. Args: stream (IO[bytes]): The stream (i.e. file-like object). Returns: int: The number of bytes. r)rfrosSEEK_END)rkcurrent_positionZ end_positionrrrrgks  rgcCs|}|dk r<|||kr(dkrs.      DUJix )9