Miscellaneous functions for manipulating text — kitchen 1.1.1 documentation

### Navigation

- index
- modules |
- next |
- previous |
- kitchen 1.1.1 documentation »
- Kitchen API »
- Kitchen.text: unicode and utf8 and xml oh my! »

# Miscellaneous functions for manipulating text

Collection of text functions that don’t fit in another category.

`kitchen.text.misc.` `byte_string_valid_encoding`(byte_string, encoding='utf-8')

  Detect if a byte `str` is valid in a specific encoding

  Parameters: | byte_string – Byte `str` to test for bytes not valid in this encoding encoding – encoding to test against. Defaults to UTF-8.
  Returns: | `True` if there are no invalid UTF-8 characters. `False` if an invalid character is detected.

  Note

  This function checks whether the byte `str` is valid in the specified encoding. It does not detect whether the byte `str` actually was encoded in that encoding. If you want that sort of functionality, you probably want to use `guess_encoding()` instead.

`kitchen.text.misc.` `byte_string_valid_xml`(byte_string, encoding='utf-8')

  Check that a byte `str` would be valid in xml

  Parameters: | byte_string – Byte `str` to check encoding – Encoding of the xml file. Default: UTF-8
  Returns: | `True` if the string is valid. `False` if it would be invalid in the xml file

  In some cases you’ll have a whole bunch of byte strings and rather than transforming them to `unicode` and back to byte `str` for output to xml, you will just want to make sure they work with the xml file you’re constructing. This function will help you do that. Example:

      ARRAY_OF_MOSTLY_UTF8_STRINGS = [...]
      processed_array = []
      for string in ARRAY_OF_MOSTLY_UTF8_STRINGS:
          if byte_string_valid_xml(string, 'utf-8'):
              processed_array.append(string)
          else:
              processed_array.append(guess_bytes_to_xml(string, encoding='utf-8'))
      output_xml(processed_array)

`kitchen.text.misc.` `guess_encoding`(byte_string, disable_chardet=False)

  Try to guess the encoding of a byte `str`

  Parameters: | byte_string – byte `str` to guess the encoding of disable_chardet – If this is True, we never attempt to use `chardet` to guess the encoding. This is useful if you need to have reproducibility whether `chardet` is installed or not. Default: `False`.
  Raises TypeError:
  | if `byte_string` is not a byte `str` type
  Returns: | string containing a guess at the encoding of `byte_string`. This is appropriate to pass as the encoding argument when encoding and decoding unicode strings.

  We start by attempting to decode the byte `str` as UTF-8. If this succeeds we tell the world it’s UTF-8 text. If it doesn’t and `chardet` is installed on the system and `disable_chardet` is False this function will use it to try detecting the encoding of `byte_string`. If it is not installed or `chardet` cannot determine the encoding with a high enough confidence then we rather arbitrarily claim that it is `latin-1`. Since `latin-1` will encode to every byte, decoding from `latin-1` to `unicode` will not cause `UnicodeErrors` although the output might be mangled.

`kitchen.text.misc.` `html_entities_unescape`(string)

  Substitute unicode characters for HTML entities

  Parameters: | string – `unicode` string to substitute out html entities
  Raises TypeError:
  | if something other than a `unicode` string is given
  Return type: | `unicode` string
  Returns: | The plain text without html entities

`kitchen.text.misc.` `process_control_chars`(string, strategy='replace')

  Look for and transform control characters in a string

  - Parameters: | string – string to search for and transform control characters within strategy – XML does not allow ASCII control characters. When we encounter those we need to know what to do. Valid options are:
    replace: | (default) Replace the control characters with `"?"`
    ignore: | Remove the characters altogether from the output
    strict: | Raise a `ControlCharError` when we encounter a control character
  Raises: | TypeError – if `string` is not a unicode string. ValueError – if the strategy is not one of replace, ignore, or strict. kitchen.text.exceptions.ControlCharError – if the strategy is `strict` and a control character is present in the `string`
  Returns: | `unicode` string with no control characters in it.

`kitchen.text.misc.` `str_eq`(str1, str2, encoding='utf-8', errors='replace')

  Compare two stringsi, converting to byte `str` if one is `unicode`

  Parameters: | str1 – First string to compare str2 – Second string to compare encoding – If we need to convert one string into a byte `str` to compare, the encoding to use. Default is utf-8. errors – What to do if we encounter errors when encoding the string. See the `kitchen.text.converters.to_bytes()` documentation for possible values. The default is `replace`.

  This function prevents `UnicodeError` (python-2.4 or less) and `UnicodeWarning` (python 2.5 and higher) when we compare a `unicode` string to a byte `str`. The errors normally arise because the conversion is done to ASCII. This function lets you convert to utf-8 or another encoding instead.

  Note

  When we need to convert one of the strings from `unicode` in order to compare them we convert the `unicode` string into a byte `str`. That means that strings can compare differently if you use different encodings for each.

  Note that `str1 == str2` is faster than this function if you can accept the following limitations:

  - Limited to python-2.5+ (otherwise a `UnicodeDecodeError` may be thrown)
  - Will generate a `UnicodeWarning` if non-ASCII byte `str` is compared to `unicode` string.

### Navigation

- index
- modules |
- next |
- previous |
- kitchen 1.1.1 documentation »
- Kitchen API »
- Kitchen.text: unicode and utf8 and xml oh my! »