Rust strings and UTF-8: parse text without broken boundaries

The first bug is often the meaning of length

A text field looks like a row of characters, but a Rust string is stored as UTF-8 bytes. Those two views coincide for plain ASCII and diverge as soon as an accented letter, another writing system or an emoji enters the input. Code that worked for every developer-written fixture can fail on a perfectly ordinary user name. The input was not exotic; the assumption was too narrow.

Start by naming the unit of each operation. A network protocol may specify a maximum byte count. A cursor may move between user-perceived characters. A parser may split at an ASCII separator. These are different requirements, so they should not share an unexplained variable called length. Rust makes byte boundaries visible because pretending that every character occupies one slot would make text handling both misleading and unsafe.

This guide uses a seven-byte example, then builds a small key-value parser. The focus is reliable application code: borrowing slices when possible, avoiding panics on external input, and writing diagnostics that distinguish missing structure from invalid values. All runnable Rust snippets use only the standard library and can be tried in Crabrix.

Inspect seven bytes before writing a parser

The string aé🦀 occupies seven UTF-8 bytes. The ASCII letter a uses one byte, é uses two, and the crab emoji uses four. It contains three Unicode scalar values. The byte offsets at valid boundaries are zero, one, three and seven. Offset two is inside the encoded é; offset four is inside the crab. A range can be in bounds numerically and still fail to describe valid string boundaries.

The hexadecimal sequence is 61 C3 A9 F0 9F A6 80. You do not need to memorize it. What matters is that byte length is a storage measurement, while scalar counting decodes the text. The example checks both measurements and demonstrates a valid and invalid slice request. get returns None for the invalid range, so the caller can decide how to respond instead of crashing.

This small exercise is valuable when investigating a reported text bug. Print the byte length, inspect char_indices, and identify the unit used by the failing operation. You can usually find the incorrect assumption more quickly than by repeatedly trying different visual strings in the interface.

fn main() {
    let text = "aé🦀";
    assert_eq!(text.len(), 7);
    assert_eq!(text.chars().count(), 3);
    assert_eq!(text.char_indices().map(|(i, _)| i).collect::<Vec<_>>(),
               vec![0, 1, 3]);
    assert_eq!(text.get(1..3), Some("é"));
    assert_eq!(text.get(1..2), None);
    println!("{} bytes, {} scalars", text.len(), text.chars().count());
}

Bytes, scalars and visible characters are different

A Rust char represents a Unicode scalar value, not an arbitrary byte and not necessarily one complete symbol on screen. An accented letter can be represented by a base letter followed by a combining mark. Emoji sequences can contain several scalars joined into one displayed symbol. Consequently, calling chars().count() does not implement a universal count of what a person sees as characters. It answers the narrower scalar-count question accurately.

This matters for truncation. Taking the first ten scalars avoids cutting a scalar’s UTF-8 encoding, but it may still separate a combining mark or part of a joined emoji sequence from the rest. A user interface that must truncate at grapheme boundaries needs an appropriate Unicode segmentation implementation. That is a separate requirement from parsing an ASCII-delimited file. Do not introduce a whole text segmentation dependency where a delimiter operation already matches the format.

The infographic deliberately labels its count as “scalars.” Use equally specific names in code and interface explanations. A byte limit should be called a byte limit. A product that promises a visible-character limit needs to define and test that promise, including combining text and the writing systems its users actually use.

Sources: The Rust Book: UTF-8 encoded strings

Use String for ownership and str for a view

String owns a growable UTF-8 buffer. &str is a borrowed view of valid UTF-8 text. A function that only reads text usually accepts &str, allowing callers to pass a string literal, a slice of an owned string, or a borrowed String. A function that must keep text independently may need an owned String. The distinction is about responsibility for storage, not about one type being the “real” string.

Borrowed parsing can avoid allocating a new string for every field. A slice returned by split_once points into the original input. That is useful while processing a line whose owner remains alive. If you store the parsed fields in a long-lived project record after the input buffer will be discarded, you need an ownership decision: copy the required fields, retain the owning buffer with an appropriate representation, or redesign the boundary. A lifetime annotation cannot make discarded bytes remain alive.

Keep APIs modest. Accept a borrowed string when reading is sufficient, and allocate at a clear point where the data must outlive its source. This makes memory behavior easier to follow and prevents scattered to_string calls from hiding which parts of the parser actually need independent storage.

Split on structure instead of guessed positions

Suppose a configuration line has the form name=value. Searching for the first equals sign expresses that format more faithfully than assuming the name occupies a certain number of bytes. split_once returns the two sides without allocating. Because the delimiter is a complete string pattern, the returned pieces are valid slices. A value can contain additional equals signs if your format assigns special meaning only to the first separator.

The function below trims surrounding whitespace, rejects a missing separator, and rejects an empty key. It permits an empty value; that is a deliberate format choice rather than an oversight. If your configuration requires nonempty values, add that rule explicitly and test it. Trimming also changes the format, so do not apply it automatically to fields where leading spaces are meaningful data.

Observe that non-ASCII text needs no special branch here. The parser does not count characters or slice at guessed offsets. It asks the string library to find the structural delimiter, then works with the resulting valid views. Choosing the right operation often removes the boundary problem entirely.

fn field(line: &str) -> Result<(&str, &str), &'static str> {
    let (key, value) = line.split_once('=').ok_or("missing = separator")?;
    let key = key.trim();
    if key.is_empty() { return Err("empty key"); }
    Ok((key, value.trim()))
}
fn main() {
    assert_eq!(field(" city = Київ "), Ok(("city", "Київ")));
    assert_eq!(field("token=a=b"), Ok(("token", "a=b")));
    assert_eq!(field("missing"), Err("missing = separator"));
    assert_eq!(field(" =value"), Err("empty key"));
    println!("All field checks passed");
}

When you need offsets, obtain them from the text

Some parsers do need byte offsets, for example to attach a diagnostic span to the original source. char_indices yields each scalar together with the byte offset where it begins. Those offsets are suitable starting boundaries for string slices. They are still byte offsets, so label them accordingly. Do not later reinterpret the same number as a screen column without a separate conversion policy.

For a range supplied by another system, use get when absence is a recoverable result. It checks both bounds and UTF-8 boundaries. Direct slicing with an invalid range panics, which can be acceptable for an internal invariant that is genuinely guaranteed, but is a poor response to an untrusted cursor position or malformed imported range. Converting failure into an explicit error lets the interface explain what went wrong.

Source spans also become stale when the underlying text changes. A correct boundary from the previous version of a document is not automatically a meaningful boundary in the next version. Associate diagnostics with the input snapshot they describe. This is especially important in editors, where a user can insert an emoji before a highlighted range while a background operation is still running.

Define an ASCII rule when the format is ASCII

A protocol identifier may intentionally allow only ASCII letters, digits and underscores. That is a valid specification if it is stated clearly. In such a field, byte-oriented validation can be appropriate because every accepted character has a single-byte encoding. The key is that you validate the restriction before relying on it. Do not infer ASCII from a few successful sample inputs and then apply byte indexing to every future string.

Human-facing names are a different domain. Rejecting a user’s name because it contains non-ASCII letters is not a necessary consequence of Rust’s string model. It is a product rule, often an undesirable one. Separate a machine identifier from a display name if they have different requirements. You can validate an internal slug narrowly while preserving the user’s original label for display.

Case conversion also needs a policy. ASCII case-insensitive comparison can be right for an ASCII keyword; it is not a complete international name comparison algorithm. Keep canonicalization requirements tied to a specific format and avoid promising that a simple lowercase operation solves every Unicode equivalence question.

Validate bytes before treating them as text

Files and network responses arrive as bytes. A String guarantees valid UTF-8, but an arbitrary byte buffer does not. If a format requires UTF-8, validate it at the boundary and return an error when decoding fails. std::str::from_utf8 provides a borrowed string view when the bytes are valid. This is different from using a lossy conversion that replaces invalid sequences with replacement characters.

Lossy conversion is useful for a diagnostic preview when showing approximate content is better than showing nothing. It is not automatically suitable for importing identifiers, verifying signatures or preserving exact source files. Replacing bytes changes the data. If you use that policy, make it visible and retain the original bytes where the application requires fidelity. A parser should not silently repair input and then claim to have accepted it unchanged.

The example makes the boundary explicit without introducing file I/O. One byte sequence decodes to é; the other is invalid UTF-8. The caller can report a decoding problem separately from a later format error such as a missing equals sign. That separation produces much more useful diagnostics.

fn main() {
    let valid = [0xC3, 0xA9];
    let invalid = [0xFF, 0x61];
    assert_eq!(std::str::from_utf8(&valid), Ok("é"));
    assert!(std::str::from_utf8(&invalid).is_err());
    println!("UTF-8 boundary checks passed");
}

Make failures describe the input rule

“Invalid string” is rarely enough information to repair an import. Distinguish decoding failures, missing separators, empty keys, duplicate keys and invalid values. Attach a line number while iterating over lines, and include a short field name when that is safe and helpful. Avoid printing entire secret-bearing configuration lines into logs just to provide context. Precision does not require exposing every byte of the input.

For numeric values, first isolate the field, then parse it into the intended numeric type, and finally check the application’s range. A value can be a valid integer and still be outside the supported interval. Reporting “sampling interval must be between the supported minimum and maximum” is more useful than treating every rejected value as a Unicode issue. Text decoding, syntactic structure and domain validation are separate stages.

This separation also helps an interface place the error correctly. A byte span can identify a source fragment, while a domain message explains the rule. Store both when you have them. Do not force a message to encode machine-readable positions in a string that another part of the application must later parse.

Build a fixture set that breaks ASCII assumptions

A useful text fixture set is small but deliberate. Include an empty string, plain ASCII, a two-byte scalar, a four-byte scalar, a combining sequence, a missing delimiter, repeated delimiters and surrounding whitespace. Add the actual writing systems and formats your product handles. For each case, state whether the expected result is acceptance, rejection or preservation without normalization. An assortment of emoji is not a substitute for a defined contract.

Test byte ranges independently from parsing rules. For aé🦀, verify that 1..3 yields é and that 1..2 is rejected. For the key-value parser, verify that an equals sign inside the value remains in the value. For imported bytes, verify that invalid UTF-8 is reported before later parsing. These assertions identify the layer responsible for a failure instead of letting one large integration test hide the cause.

If the interface truncates names, add a visual check for combined and joined text alongside automated boundary tests. A string can remain valid UTF-8 while looking broken to a reader. Correct storage encoding is a foundation, but readable presentation requires the additional rules chosen by your interface.

Choose the line-ending contract too

A line-oriented importer also needs a policy for line endings. Rust’s lines method treats a newline and a carriage-return/newline pair as line terminators, and does not return a final empty line merely because the input ends with a terminator. Splitting literally on \n has different behavior: it can leave a carriage return attached to a field and can yield a trailing empty piece. Neither operation is a universal replacement for the other.

For a configuration reader, lines is often convenient. For a format where an empty final record is significant, inspect the format before choosing it. Add fixtures for Unix-style endings, Windows-style endings and a final terminator. This is another example of selecting an operation from the contract rather than assuming that every visually similar text file has identical structure.

A compact plan for safer text handling

At each boundary, ask four questions: who owns the bytes, are they valid UTF-8, what structure does the format require, and what unit does each length or position use? Most accidental string bugs involve an unstated answer to one of these questions. Writing down the answers often leads directly to a suitable standard-library method and a much smaller implementation.

Prefer operations that match the format: delimiter splitting for delimited fields, lines for line-oriented text, get for a fallible byte range, and char_indices when decoded scalar boundaries matter. Keep owned copies for places that need independent storage. Reserve grapheme-aware processing for requirements about user-perceived characters, and use a suitable implementation rather than assuming chars already provides it.

The next time a text bug appears only with one user’s input, resist calling that input unusual. Reduce it to a small fixture, identify the mistaken unit or ownership assumption, and keep the fixture as a regression test. Rust’s explicit string model gives you the tools to explain the failure and repair the rule rather than patching individual characters.

Sources: Standard library: str methods and UTF-8 guarantees